clustpy.deep.autoencoders package

Submodules

clustpy.deep.autoencoders.convolutional_autoencoder module

class clustpy.deep.autoencoders.convolutional_autoencoder.ConvolutionalAutoencoder(input_height: int, fc_layers: list, conv_encoder_name: str = 'resnet18', conv_decoder_name: str | None = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.ReLU'>, fc_decoder_layers: list | None = None, decoder_output_fn: ~torch.nn.modules.module.Module | None = None, pretrained_encoder_weights: ~torchvision.models._api.Weights | None = None, pretrained_decoder_weights: ~torchvision.models._api.Weights | None = None, reusable: bool = True, **fc_kwargs)

Bases: _AbstractAutoencoder

A convolutional autoencoder based on the ResNet architecture.

Parameters:

• input_height (int) – height of the images for the decoder (assume square images)

• fc_layers (list) – list of the different layer sizes from flattened convolutional layer input to embedding. First input needs to be 512 if conv_encoder_name=”resnet18” and 2048 if conv_encoder_name=”resnet50”. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.

• conv_encoder_name (str) – name of convolutional resnet encoder part of the autoencoder. Can be ‘resnet18’ or ‘resnet50’ (default: ‘resnet18’)

• conv_decoder_name (str) – name of convolutional resnet decoder part of the autoencoder. Can be ‘resnet18’ or ‘resnet50’. If None it will be the same as conv_encoder_name (default: None)

• activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)

• fc_decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)

• decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)

• pretrained_encoder_weights (torchvision.models._api.Weights) – weights from torch.vision.models, indicates whether pretrained resnet weights should be used for the encoder. (default: None)

• pretrained_decoder_weights (torchvision.models._api.Weights) – weights from torch.vision.models, indicates whether pretrained resnet weights should be used for the decoder (not implemented yet). (default: None)

• reusable (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)

• fc_kwargs (dict) – additional parameters for FullyConnectedBlock

conv_encoder

convolutional resnet encoder part of the autoencoder

Type:

ResNetEncoder

conv_decoder

convolutional resnet decoder part of the autoencoder

Type:

ResNetEncoder

fc_encoder

fully connected encoder part of the convolutional autoencoder, together with conv_encoder is responsible for embedding data points

Type:

FullyConnectedBlock

fc_decoder

fully connected decoder part of the convolutional autoencoder, together with conv_decoder is responsible for reconstructing data points from the embedding

Type:

FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:

bool

reusable

indicates whether the autoencoder should be reused by multiple deep clustering algorithms

Type:

bool

References

He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

and

LeCun, Yann, et al. “Backpropagation applied to handwritten zip code recognition.” Neural computation 1.4 (1989): 541-551.

decode(embedded: Tensor) → Tensor

Apply the decoder function to embedded. Runs x through the conv_decoder and then the fc_decoder.

Parameters:

embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points

Returns:

decoded – returns the reconstruction of embedded

Return type:

torch.Tensor

encode(x: Tensor) → Tensor

Apply the encoder function to x. Runs x through the conv_encoder and then the fc_encoder.

Parameters:

x (torch.Tensor) – input data point, can also be a mini-batch of points

Returns:

embedded – the embedded data point with dimensionality embedding_size

Return type:

torch.Tensor

training: bool

clustpy.deep.autoencoders.feedforward_autoencoder module

@authors: Lukas Miklautz

class clustpy.deep.autoencoders.feedforward_autoencoder.FeedforwardAutoencoder(layers: list, batch_norm: bool = False, dropout: float | None = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list | None = None, decoder_output_fn: ~torch.nn.modules.module.Module | None = None, reusable: bool = True)

Bases: _AbstractAutoencoder

A flexible feedforward autoencoder.

Parameters:

• layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.

• batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)

• dropout (float) – Set the amount of dropout you want to use (default: None)

• activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)

• bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)

• decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)

• decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)

• reusable (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:

bool

reusable

indicates whether the autoencoder should be reused by multiple deep clustering algorithms

Type:

bool

References

E.g. Ballard, Dana H. “Modular learning in neural networks.” Aaai. Vol. 647. 1987.

decode(embedded: Tensor) → Tensor

Apply the decoder function to embedded.

Parameters:

embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points

Returns:

decoded – returns the reconstruction of embedded

Return type:

torch.Tensor

encode(x: Tensor) → Tensor

Apply the encoder function to x.

Parameters:

x (torch.Tensor) – input data point, can also be a mini-batch of points

Returns:

embedded – the embedded data point with dimensionality embedding_size

Return type:

torch.Tensor

training: bool

clustpy.deep.autoencoders.neighbor_encoder module

@authors: Collin Leiber

class clustpy.deep.autoencoders.neighbor_encoder.NeighborEncoder(layers: list, n_neighbors: int, decode_self: bool = False, batch_norm: bool = False, dropout: float | None = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list | None = None, decoder_output_fn: ~torch.nn.modules.module.Module | None = None, reusable: bool = True)

Bases: FeedforwardAutoencoder

A NeighborEncoder. Does not compare the reconstruction of an object to itself but to its nearest neighbors. For more information see the stated reference. If n_neighbors is 0 and decode_self is true, the NeighborEncoder will work as a regular FeedforwardAutoencoder.

Parameters:

• layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.

• n_neighbors (int) – the number of nearest neighbors to be considered

• decode_self (bool) – specifies whether a point itself should also be decoded (default: False)

• batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)

• dropout (float) – Set the amount of dropout you want to use (default: None)

• activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)

• bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)

• decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)

• decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)

• reusable (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing the data point itself (class is FullyConnectedBlock). Only used if decode_self is true.

Type:

FullyConnectedBlock

neighbor_decoders

list containing one decoder network (class is FullyConnectedBlock) for each nearest neighbor

Type:

list

fitted

boolean value indicating whether the autoencoder is already fitted

Type:

bool

reusable

indicates whether the autoencoder should be reused by multiple deep clustering algorithms

Type:

bool

Examples

>>> from clustpy.data import create_subspace_data
>>> from clustpy.deep import get_dataloader
>>> from clustpy.deep._utils import detect_device
>>> from scipy.spatial.distance import pdist, squareform

>>> X, L = create_subspace_data(1500, subspace_features=(3, 50), random_state=1)
>>> device = detect_device()
>>> n_neighbors = 3

>>> dist_matrix = squareform(pdist(X))
>>> neighbor_ids = np.argsort(dist_matrix, axis=1)
>>> neighbors = [X[neighbor_ids[:, 1 + i]] for i in range(n_neighbors)]
>>> # Alternatively: neighbors = get_neighbors_batchwise(X, n_neighbors)

>>> dataloader = get_dataloader(X, 256, True, additional_inputs=neighbors)
>>> neighbor_encoder = NeighborEncoder(layers=[X.shape[1], 512, 256, 10], n_neighbors=n_neighbors, decode_self=False)
>>> neighbor_encoder.fit(dataloader=dataloader, device=device, n_epochs=5, lr=1e-3)

References

Yeh, Chin-Chia Michael, et al. “Representation Learning by Reconstructing Neighborhoods.” arXiv preprint arXiv:1811.01557 (2018).

decode(embedded: Tensor) → Tensor

Apply the decoder function of each neighbor network to embedded. Returns a (n_neighbors x batch_size x dimensionality) tensor if decode_self is false, else a (n_neighbors + 1 x batch_size x dimensionality) tensor

Parameters:

embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points

Returns:

decoded_neighbors – returns the reconstructions of the embedded sample concerning its neighbor decoders

Return type:

torch.Tensor

fit(n_epochs: int, optimizer_params: dict, batch_size: int = 128, dataloader: ~torch.utils.data.dataloader.DataLoader = None, evalloader: ~torch.utils.data.dataloader.DataLoader = None, optimizer_class: ~torch.optim.optimizer.Optimizer = <class 'torch.optim.adam.Adam'>, loss_fn: ~torch.nn.modules.loss._Loss = MSELoss(), patience: int = 5, scheduler: <module 'torch.optim.lr_scheduler' from '/home/docs/checkouts/readthedocs.org/user_builds/clustpy/envs/v0.0.2-beta/lib/python3.8/site-packages/torch/optim/lr_scheduler.py'> = None, scheduler_params: dict = None, device: ~torch.device = device(type='cpu'), model_path: str = None, print_step: int = 0) → NeighborEncoder

Trains the NeighborEncoder in place. Equal to fit function of the FeedforwardAutoencoder but does only work with a dataloader (not with a regular data array). This is because the dataloader must contain the nearest neighbors of each point at the positions 2, 3, ….

Parameters:

• n_epochs (int) – number of epochs for training

• optimizer_params (dict) – parameters of the optimizer, includes the learning rate

• batch_size (int) – size of the data batches (default: 128)

• dataloader (torch.utils.data.DataLoader) – dataloader to be used for training (default: default=None)

• evalloader (torch.utils.data.DataLoader) – dataloader to be used for evaluation, early stopping and learning rate scheduling if scheduler=torch.optim.lr_scheduler.ReduceLROnPlateau (default: None)

• optimizer_class (torch.optim.Optimizer) – optimizer to be used (default: torch.optim.Adam)

• loss_fn (torch.nn.modules.loss._Loss) – loss function to be used for reconstruction (default: torch.nn.MSELoss())

• patience (int) – patience parameter for EarlyStopping (default: 5)

• scheduler (torch.optim.lr_scheduler) – learning rate scheduler that should be used. If torch.optim.lr_scheduler.ReduceLROnPlateau is used then the behaviour is matched by providing the validation_loss calculated based on samples from evalloader (default: None)

• scheduler_params (dict) – dictionary of the parameters of the scheduler object (default: None)

• device (torch.device) – device to be trained on (default: torch.device(‘cpu’))

• model_path (str) – if specified will save the trained model to the location. If evalloader is used, then only the best model w.r.t. evaluation loss is saved (default: None)

• print_step (int) – specifies how often the losses are printed. If 0, no prints will occur (default: 0)

Returns:

self – this instance of the NeighborEncoder

Return type:

NeighborEncoder

loss(batch: list, loss_fn: ~torch.nn.modules.loss._Loss, device: ~torch.device) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)

Calculate the loss of a single batch of data. Corresponds to the sum of losses concerning each neighbor. batch must contain the data object at the first position and the neighbors at the following positions.

Parameters:

• batch (list) – the different parts of a dataloader (id, samples, 1-nearest-neighbor, 2-nearest-neighbor, …)

• loss_fn (torch.nn.modules.loss._Loss) – loss function to be used for reconstruction

• device (torch.device) – device to be trained on

Returns:

loss – the sum of the reconstruction losses of the input sample, the embedded input sample, the reconstructions of the embedded sample concerning its neighbor decoders

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor)

training: bool

clustpy.deep.autoencoders.neighbor_encoder.get_neighbors_batchwise(X: ndarray, n_neighbors: int, metric: str = 'sqeuclidean', batch_size: int = 10000) → list

For large datasets it is often not possible to determine the nearest neighbors in a trivial manner. Therefore, here is an implementation that calculates the nearest neighbors in batches. Ignores the objects themselves (with distance of 0) as nearest neighbors. It reduces the memory consumption of a trivial nearest neighbor implementation from (data_size x data_size) to (batch_size x data_size). A list is returned, which can be given as additional input into a DataLoader and is therefore directly compatible with the NeighborEncoder. Due to runtime concerns it is still recommended to use a more complex nearest neighbor retrieval implementation (e.g. from sklearn.neighbor)!

Parameters:

• X (np.ndarray) – The given data set

• n_neighbors (int) – The number of nearest neighbors to identify

• metric (str) – The distance metric to be used. See scipy.spatial.distance.cdist for more information (default: sqeuclidean)

• batch_size (int) – The size of the batches (default: 10000)

Returns:

nearest_neigbors – A list containing the nearest neighbors as torch.Tensors, i.e. [1-nearest-neighbor tensor, 2-nearest-neighbor tensor, …]

Return type:

list

Examples

>>> from clustpy.data import create_subspace_data
>>> from clustpy.deep import get_dataloader
>>> X, L = create_subspace_data(1500, subspace_features=(3, 50), random_state=1)
>>> n_neighbors = 3
>>> neighbors = get_neighbors_batchwise(X, n_neighbors)
>>> dataloader = get_dataloader(X, 256, True, additional_inputs=neighbors)
>>> neighbor_encoder = NeighborEncoder(layers=[X.shape[1], 512, 256, 10], n_neighbors=n_neighbors)
>>> neighbor_encoder.fit(dataloader=dataloader, n_epochs=5, lr=1e-3)

clustpy.deep.autoencoders.stacked_autoencoder module

@authors: Dominik Mautz

class clustpy.deep.autoencoders.stacked_autoencoder.StackedAutoencoder(feature_dim, embedding_size, layer_dims=[500, 500, 2000], weight_initalizer=<function xavier_normal_>, activation_fn=<function StackedAutoencoder.<lambda>>, loss_fn=<function StackedAutoencoder.<lambda>>, optimizer_fn=<function StackedAutoencoder.<lambda>>, tied_weights=False, bias_init=0.0, linear_embedded=True, linear_decoder_last=True)

Bases: Module

decode(encoded_data)

encode(input_data)

forward(input_data)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

forward_pretrain(input_data, stack, use_dropout=True, dropout_rate=0.2, dropout_is_training=True)

parameters_pretrain(stack)

pretrain(dataset, device, rounds_per_layer=1000, dropout_rate=0.2, corruption_fn=None)

Uses Adam to pretrain the model layer by layer :param rounds_per_layer: :param corruption_fn: Can be used to corrupt the input data for an denoising autoencoder :return:

refine_training(dataset, rounds, device, corruption_fn=None, optimizer_fn=None)

start_training(trainloader, device, steps_per_layer=10000, refine_training_steps=20000, dropout_rate=0.2)

training: bool

clustpy.deep.autoencoders.variational_autoencoder module

@authors: Lukas Miklautz, Donatella Novakovic, Collin Leiber

class clustpy.deep.autoencoders.variational_autoencoder.VariationalAutoencoder(layers: list, batch_norm: bool = False, dropout: float | None = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list | None = None, decoder_output_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.Sigmoid'>, reusable: bool = True)

Bases: FeedforwardAutoencoder

A variational autoencoder (VAE).

Parameters:

• layers (list) –

  list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the dimension of the mean and variance value in the central layer.

  If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.

• batch_norm (bool) – set True if you want to use torch.nn.BatchNorm1d (default: False)

• dropout (float) – set the amount of dropout you want to use (default: None)

• activation (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)

• bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)

• decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)

• decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: torch.nn.Sigmoid)

• reusable (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

mean

mean value of the central layer

Type:

torch.nn.Linear

log_variance

logarithmic variance of the central layer (use logarithm of variance - numerical purposes)

Type:

torch.nn.Linear

fitted

boolean value indicating whether the autoencoder is already fitted

Type:

bool

reusable

indicates whether the autoencoder should be reused by mutliple deep clustering algorithms

Type:

bool

References

Kingma, Diederik P., and Max Welling. “Auto-encoding variational Bayes.” Int. Conf. on Learning Representations.

encode(x: ~torch.Tensor) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>)

Apply the encoder function to x. Overwrites function from FeedforwardAutoencoder.

Parameters:

x (torch.Tensor) – input data point, can also be a mini-batch of points

Returns:

tuple – mean value of the central VAE layer, logarithmic variance value of the central VAE layer (use logarithm of variance - numerical purposes)

Return type:

(torch.Tensor, torch.Tensor)

forward(x: ~torch.Tensor) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)

Applies both the encode and decode function. The forward function is automatically called if we call self(x). Overwrites function from FeedforwardAutoencoder.

Parameters:

x (torch.Tensor) – input data point, can also be a mini-batch of embedded points

Returns:

tuple – sampling using q_mean and q_logvar, mean value of the central VAE layer, logarithmic variance value of the central VAE layer (use logarithm of variance - numerical purposes), the reconstruction of the data point

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor)

loss(batch: list, loss_fn: ~torch.nn.modules.loss._Loss, device: ~torch.device, beta: float = 1) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)

Calculate the loss of a single batch of data.

Parameters:

• batch (list) – the different parts of a dataloader (id, samples, …)

• loss_fn (torch.nn.modules.loss._Loss) – loss function to be used for reconstruction

• device (torch.device) – device to be trained on

• beta (float) – weighting of the KL loss (default: 1)

Returns:

total_loss – the reconstruction loss of the input sample, the sampling, the reconstruction of the data point

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor)

training: bool

Module contents

class clustpy.deep.autoencoders.ConvolutionalAutoencoder(input_height: int, fc_layers: list, conv_encoder_name: str = 'resnet18', conv_decoder_name: str | None = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.ReLU'>, fc_decoder_layers: list | None = None, decoder_output_fn: ~torch.nn.modules.module.Module | None = None, pretrained_encoder_weights: ~torchvision.models._api.Weights | None = None, pretrained_decoder_weights: ~torchvision.models._api.Weights | None = None, reusable: bool = True, **fc_kwargs)

Bases: _AbstractAutoencoder

A convolutional autoencoder based on the ResNet architecture.

Parameters:

• input_height (int) – height of the images for the decoder (assume square images)

• fc_layers (list) – list of the different layer sizes from flattened convolutional layer input to embedding. First input needs to be 512 if conv_encoder_name=”resnet18” and 2048 if conv_encoder_name=”resnet50”. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.

• conv_encoder_name (str) – name of convolutional resnet encoder part of the autoencoder. Can be ‘resnet18’ or ‘resnet50’ (default: ‘resnet18’)

• conv_decoder_name (str) – name of convolutional resnet decoder part of the autoencoder. Can be ‘resnet18’ or ‘resnet50’. If None it will be the same as conv_encoder_name (default: None)

• activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)

• fc_decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)

• decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)

• pretrained_encoder_weights (torchvision.models._api.Weights) – weights from torch.vision.models, indicates whether pretrained resnet weights should be used for the encoder. (default: None)

• pretrained_decoder_weights (torchvision.models._api.Weights) – weights from torch.vision.models, indicates whether pretrained resnet weights should be used for the decoder (not implemented yet). (default: None)

• reusable (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)

• fc_kwargs (dict) – additional parameters for FullyConnectedBlock

conv_encoder

convolutional resnet encoder part of the autoencoder

Type:

ResNetEncoder

conv_decoder

convolutional resnet decoder part of the autoencoder

Type:

ResNetEncoder

fc_encoder

fully connected encoder part of the convolutional autoencoder, together with conv_encoder is responsible for embedding data points

Type:

FullyConnectedBlock

fc_decoder

fully connected decoder part of the convolutional autoencoder, together with conv_decoder is responsible for reconstructing data points from the embedding

Type:

FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:

bool

reusable

indicates whether the autoencoder should be reused by multiple deep clustering algorithms

Type:

bool

References

He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

and

LeCun, Yann, et al. “Backpropagation applied to handwritten zip code recognition.” Neural computation 1.4 (1989): 541-551.

decode(embedded: Tensor) → Tensor

Apply the decoder function to embedded. Runs x through the conv_decoder and then the fc_decoder.

Parameters:

embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points

Returns:

decoded – returns the reconstruction of embedded

Return type:

torch.Tensor

encode(x: Tensor) → Tensor

Apply the encoder function to x. Runs x through the conv_encoder and then the fc_encoder.

Parameters:

x (torch.Tensor) – input data point, can also be a mini-batch of points

Returns:

embedded – the embedded data point with dimensionality embedding_size

Return type:

torch.Tensor

training: bool

class clustpy.deep.autoencoders.FeedforwardAutoencoder(layers: list, batch_norm: bool = False, dropout: float | None = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list | None = None, decoder_output_fn: ~torch.nn.modules.module.Module | None = None, reusable: bool = True)

Bases: _AbstractAutoencoder

A flexible feedforward autoencoder.

Parameters:

• layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.

• batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)

• dropout (float) – Set the amount of dropout you want to use (default: None)

• activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)

• bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)

• decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)

• decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)

• reusable (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:

bool

reusable

indicates whether the autoencoder should be reused by multiple deep clustering algorithms

Type:

bool

References

E.g. Ballard, Dana H. “Modular learning in neural networks.” Aaai. Vol. 647. 1987.

decode(embedded: Tensor) → Tensor

Apply the decoder function to embedded.

Parameters:

embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points

Returns:

decoded – returns the reconstruction of embedded

Return type:

torch.Tensor

encode(x: Tensor) → Tensor

Apply the encoder function to x.

Parameters:

x (torch.Tensor) – input data point, can also be a mini-batch of points

Returns:

embedded – the embedded data point with dimensionality embedding_size

Return type:

torch.Tensor

training: bool

class clustpy.deep.autoencoders.NeighborEncoder(layers: list, n_neighbors: int, decode_self: bool = False, batch_norm: bool = False, dropout: float | None = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list | None = None, decoder_output_fn: ~torch.nn.modules.module.Module | None = None, reusable: bool = True)

Bases: FeedforwardAutoencoder

A NeighborEncoder. Does not compare the reconstruction of an object to itself but to its nearest neighbors. For more information see the stated reference. If n_neighbors is 0 and decode_self is true, the NeighborEncoder will work as a regular FeedforwardAutoencoder.

Parameters:

• layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.

• n_neighbors (int) – the number of nearest neighbors to be considered

• decode_self (bool) – specifies whether a point itself should also be decoded (default: False)

• batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)

• dropout (float) – Set the amount of dropout you want to use (default: None)

• activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)

• bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)

• decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)

• decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)

• reusable (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing the data point itself (class is FullyConnectedBlock). Only used if decode_self is true.

Type:

FullyConnectedBlock

neighbor_decoders

list containing one decoder network (class is FullyConnectedBlock) for each nearest neighbor

Type:

list

fitted

boolean value indicating whether the autoencoder is already fitted

Type:

bool

reusable

indicates whether the autoencoder should be reused by multiple deep clustering algorithms

Type:

bool

Examples

>>> from clustpy.data import create_subspace_data
>>> from clustpy.deep import get_dataloader
>>> from clustpy.deep._utils import detect_device
>>> from scipy.spatial.distance import pdist, squareform

>>> X, L = create_subspace_data(1500, subspace_features=(3, 50), random_state=1)
>>> device = detect_device()
>>> n_neighbors = 3

>>> dist_matrix = squareform(pdist(X))
>>> neighbor_ids = np.argsort(dist_matrix, axis=1)
>>> neighbors = [X[neighbor_ids[:, 1 + i]] for i in range(n_neighbors)]
>>> # Alternatively: neighbors = get_neighbors_batchwise(X, n_neighbors)

>>> dataloader = get_dataloader(X, 256, True, additional_inputs=neighbors)
>>> neighbor_encoder = NeighborEncoder(layers=[X.shape[1], 512, 256, 10], n_neighbors=n_neighbors, decode_self=False)
>>> neighbor_encoder.fit(dataloader=dataloader, device=device, n_epochs=5, lr=1e-3)

References

Yeh, Chin-Chia Michael, et al. “Representation Learning by Reconstructing Neighborhoods.” arXiv preprint arXiv:1811.01557 (2018).

decode(embedded: Tensor) → Tensor

Apply the decoder function of each neighbor network to embedded. Returns a (n_neighbors x batch_size x dimensionality) tensor if decode_self is false, else a (n_neighbors + 1 x batch_size x dimensionality) tensor

Parameters:

embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points

Returns:

decoded_neighbors – returns the reconstructions of the embedded sample concerning its neighbor decoders

Return type:

torch.Tensor

fit(n_epochs: int, optimizer_params: dict, batch_size: int = 128, dataloader: ~torch.utils.data.dataloader.DataLoader = None, evalloader: ~torch.utils.data.dataloader.DataLoader = None, optimizer_class: ~torch.optim.optimizer.Optimizer = <class 'torch.optim.adam.Adam'>, loss_fn: ~torch.nn.modules.loss._Loss = MSELoss(), patience: int = 5, scheduler: <module 'torch.optim.lr_scheduler' from '/home/docs/checkouts/readthedocs.org/user_builds/clustpy/envs/v0.0.2-beta/lib/python3.8/site-packages/torch/optim/lr_scheduler.py'> = None, scheduler_params: dict = None, device: ~torch.device = device(type='cpu'), model_path: str = None, print_step: int = 0) → NeighborEncoder

Trains the NeighborEncoder in place. Equal to fit function of the FeedforwardAutoencoder but does only work with a dataloader (not with a regular data array). This is because the dataloader must contain the nearest neighbors of each point at the positions 2, 3, ….

Parameters:

• n_epochs (int) – number of epochs for training

• optimizer_params (dict) – parameters of the optimizer, includes the learning rate

• batch_size (int) – size of the data batches (default: 128)

• dataloader (torch.utils.data.DataLoader) – dataloader to be used for training (default: default=None)

• evalloader (torch.utils.data.DataLoader) – dataloader to be used for evaluation, early stopping and learning rate scheduling if scheduler=torch.optim.lr_scheduler.ReduceLROnPlateau (default: None)

• optimizer_class (torch.optim.Optimizer) – optimizer to be used (default: torch.optim.Adam)

• loss_fn (torch.nn.modules.loss._Loss) – loss function to be used for reconstruction (default: torch.nn.MSELoss())

• patience (int) – patience parameter for EarlyStopping (default: 5)

• scheduler (torch.optim.lr_scheduler) – learning rate scheduler that should be used. If torch.optim.lr_scheduler.ReduceLROnPlateau is used then the behaviour is matched by providing the validation_loss calculated based on samples from evalloader (default: None)

• scheduler_params (dict) – dictionary of the parameters of the scheduler object (default: None)

• device (torch.device) – device to be trained on (default: torch.device(‘cpu’))

• model_path (str) – if specified will save the trained model to the location. If evalloader is used, then only the best model w.r.t. evaluation loss is saved (default: None)

• print_step (int) – specifies how often the losses are printed. If 0, no prints will occur (default: 0)

Returns:

self – this instance of the NeighborEncoder

Return type:

NeighborEncoder

loss(batch: list, loss_fn: ~torch.nn.modules.loss._Loss, device: ~torch.device) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)

Calculate the loss of a single batch of data. Corresponds to the sum of losses concerning each neighbor. batch must contain the data object at the first position and the neighbors at the following positions.

Parameters:

• batch (list) – the different parts of a dataloader (id, samples, 1-nearest-neighbor, 2-nearest-neighbor, …)

• loss_fn (torch.nn.modules.loss._Loss) – loss function to be used for reconstruction

• device (torch.device) – device to be trained on

Returns:

loss – the sum of the reconstruction losses of the input sample, the embedded input sample, the reconstructions of the embedded sample concerning its neighbor decoders

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor)

training: bool

class clustpy.deep.autoencoders.StackedAutoencoder(feature_dim, embedding_size, layer_dims=[500, 500, 2000], weight_initalizer=<function xavier_normal_>, activation_fn=<function StackedAutoencoder.<lambda>>, loss_fn=<function StackedAutoencoder.<lambda>>, optimizer_fn=<function StackedAutoencoder.<lambda>>, tied_weights=False, bias_init=0.0, linear_embedded=True, linear_decoder_last=True)

Bases: Module

decode(encoded_data)

encode(input_data)

forward(input_data)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

forward_pretrain(input_data, stack, use_dropout=True, dropout_rate=0.2, dropout_is_training=True)

parameters_pretrain(stack)

pretrain(dataset, device, rounds_per_layer=1000, dropout_rate=0.2, corruption_fn=None)

Uses Adam to pretrain the model layer by layer :param rounds_per_layer: :param corruption_fn: Can be used to corrupt the input data for an denoising autoencoder :return:

refine_training(dataset, rounds, device, corruption_fn=None, optimizer_fn=None)

start_training(trainloader, device, steps_per_layer=10000, refine_training_steps=20000, dropout_rate=0.2)

training: bool

class clustpy.deep.autoencoders.VariationalAutoencoder(layers: list, batch_norm: bool = False, dropout: float | None = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list | None = None, decoder_output_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.Sigmoid'>, reusable: bool = True)

Bases: FeedforwardAutoencoder

A variational autoencoder (VAE).

Parameters:

• layers (list) –

  list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the dimension of the mean and variance value in the central layer.

  If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.

• batch_norm (bool) – set True if you want to use torch.nn.BatchNorm1d (default: False)

• dropout (float) – set the amount of dropout you want to use (default: None)

• activation (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)

• bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)

• decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)

• decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: torch.nn.Sigmoid)

• reusable (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:

FullyConnectedBlock

mean

mean value of the central layer

Type:

torch.nn.Linear

log_variance

logarithmic variance of the central layer (use logarithm of variance - numerical purposes)

Type:

torch.nn.Linear

fitted

boolean value indicating whether the autoencoder is already fitted

Type:

bool

reusable

indicates whether the autoencoder should be reused by mutliple deep clustering algorithms

Type:

bool

References

Kingma, Diederik P., and Max Welling. “Auto-encoding variational Bayes.” Int. Conf. on Learning Representations.

encode(x: ~torch.Tensor) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>)

Apply the encoder function to x. Overwrites function from FeedforwardAutoencoder.

Parameters:

x (torch.Tensor) – input data point, can also be a mini-batch of points

Returns:

tuple – mean value of the central VAE layer, logarithmic variance value of the central VAE layer (use logarithm of variance - numerical purposes)

Return type:

(torch.Tensor, torch.Tensor)

forward(x: ~torch.Tensor) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)

Applies both the encode and decode function. The forward function is automatically called if we call self(x). Overwrites function from FeedforwardAutoencoder.

Parameters:

x (torch.Tensor) – input data point, can also be a mini-batch of embedded points

Returns:

tuple – sampling using q_mean and q_logvar, mean value of the central VAE layer, logarithmic variance value of the central VAE layer (use logarithm of variance - numerical purposes), the reconstruction of the data point

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor)

loss(batch: list, loss_fn: ~torch.nn.modules.loss._Loss, device: ~torch.device, beta: float = 1) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)

Calculate the loss of a single batch of data.

Parameters:

• batch (list) – the different parts of a dataloader (id, samples, …)

• loss_fn (torch.nn.modules.loss._Loss) – loss function to be used for reconstruction

• device (torch.device) – device to be trained on

• beta (float) – weighting of the KL loss (default: 1)

Returns:

total_loss – the reconstruction loss of the input sample, the sampling, the reconstruction of the data point

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor)

training: bool