Distributed Bloom Filter

Target audience: Intermediate

Estimated reading time: 5'

Posts history

                Have you ever been in need of a method that's both effective and has low latency for checking if a particular object or piece of data belongs to a significantly large dataset? 

In this article, we're going to explore the concept of a distributed Bloom filter, utilizing Apache Spark and the application of a cryptographic digest as well as estimate the false positives.

Table of contents

Overview

Theory

Digest-based filter

       Scala implementation

       Use case

       Performance evaluation

Spark-based filter

       Implementation

       Impact of capacity on expected false positives

       Application to datasets

References

What you will learn: Creating and analyzing a Bloom filter for processing extremely large datasets with Apache Spark and cryptographic hashes, with a focus on understanding and managing false positives.

Notes: 

• For the sake of readability of the implementation of algorithms, all non-essential code such as error checking, comments, exception, validation of class and method arguments, scoping qualifiers or import is omitted 
• The code associated with this article is written using Scala 2.12.4 and Apache Spark 3.4.0
• Source code available on GitHub GitHub Patrick Nicolas Bloom Filter

Overview

Bloom filter became a popular probabilistic data structure to enable membership queries (object x belonging to set or category Y) a couple of years ago. The main benefit of Bloom filter is to reduce the requirement of large memory allocation by avoiding allocating objects in memory much like HashSet or HashMap. The compact representation comes with a trade-off: although the filter does not allow false negatives it does not guarantee that there are no false positives. 

In other words, a query returns:

• very high probability that an object belongs to a set
• an object does not belong to a set.

A Bloom filter is quite often used as a front end to a deterministic algorithm

Theory

Let's consider a set A = {a0,.. an-1} of n elements for which a query to determine membership is executed. The data structure consists of a bit vector V of m bits and k completely independent hash functions that are associated to a position in the bit vector. The assignment (or mapping) of hash functions to bits has to follow a uniform distribution [ref 1].

The diagram below illustrates the basic mechanism behind the Bloom filter. The set A is defined by the pair a1 and a2. The hash functions h1 and h2 map the elements to bit position (bit set to 1) in the bit vector. The element b has one of the positions set to 0 and therefore does not belong to the set. The element c belongs to the set because its associated positions have bits set to 1

However, the algorithm does not prevent false positive. For instance, a bit may have been set to 1 during the insertion of previous elements and the query reports erroneously that the element belongs to the set.

The insertion of an elements depends on the h hash functions, therefore the time needed to add a new element is h (number of hash functions) and independent from size of the bit vector: asymptotic insertion time = O(h). However, the filter requires h bits for each element and is less effective that traditional bit array for small sets.

The probability of false positives decreases as the number n of inserted elements decreases and the size of the bitvector m, increases. The number of hash functions that minimizes the probability of false positives is defined by h = m.ln2/n.

Digest-based filter

Scala implementation

The approach utilizes cryptographic hash functions, referencing both [ref 2] and employing the MessageDigest class from the Java library [ref 3] to create unique hash codes. Details such as auxiliary methods and specific conditions for method parameters are excluded for simplicity. 

The initial step involves establishing the DigestBloomFilter class along with its properties:

• length Number of entries in the filter
• numHashFunctions Number of hash functions
• hashingAlgo Hashing algorithm with SHA1 as default
• set Array of bytes for entries in the Bloom filter
• digest Digest used to generate hash values

class DigestBloomFilter[T: ClassTag](
  length: Int,             // Length or capacity of the Bloom filter
  numHashFunctions: Int,   // Number of hash functions
  hashingAlgo: HashingAlgo = SHA1Algo()  // Hashing algorithm SHA1, MD5, ..
) extends BloomFilter[T] {

  private[this] val set: Array[Byte] = new Array[Byte](length)
  private[this] val digest = Try(MessageDigest.getInstance(hashingAlgo.toString))
  private[this] var size: Int = 0

  // Add a new element of type T to the set of the Bloom filter
  override def add(t: T): Unit = {
     hashToArray(t).foreach(set(_) = 1)
     size += 1
  }

  // Add an array of elements of type T to the filter
  override def addAll(ts: Array[T]): Unit =
     if(ts.nonEmpty)
       digest.foreach(_ => ts.foreach(add))

   // Test whether the filter might contain a given element
  override def mightContain(t: T): Boolean =
     digest.map(_ => hashToArray(t).forall(set(_) == 1)).getOrElse(false)

The digest using the message digest of the java library java.security.MessageDigest.

The next step consists of defining the methods to add single generic element add(t: T) and array of elements addAll(ts: Array[T])

The method mightContain evaluates whether an element is contained in the filter. The method returns

• true if the filter very likely contains the element
• false if the filter DOES NOT contain this element

The add and mightContain methods relies on the hashToArray private method to initialize the set of entries, with the first value being the hashCode of the new entry.

def hashToArray(t: T): Array[Int] = 
   (0 until numHashFunctions).foldLeft(new Array[Int](numHashFunctions))(
     (buf, idx) => {
       val value = if(idx > 0) hash(buf(idx -1)) else hash(t.hashCode)
       buf.update(idx, value)
       buf
     }
  )

The hash method is the core of the Bloom filter: It consists of computing an index of an entry.

def hash(value: Int): Int = digest.map(
  d => {
    d.reset()
    d.update(value)
    Math.abs(new BigInteger(1, d.digest).intValue) % (set.length - 1)
  }
).getOrElse(-1)

The instance of the MessageDigest class, digest generates a hash value using either MD5 or SHA-1 algorithm. Tail recursion is used as an alternative to the iterative process to generate the set. 

The next code snippet implements a very simple implicit conversion from Int to Array[Byte] conversion 

val numBytes: Int = 4
val lastByte: Int = numBytes - 1

implicit def int2Bytes(value: Int): Array[Byte] = Array.tabulate(numBytes)(
    n => {
      val offset = (lastByte - n) << lastByte
      ((value >>> offset) & 0xFF).toByte
    }
  )

The conversion relies on the manipulation of bits from a 32 bit Integer to 4 bytes. Alternatively, you may consider a conversion from a long value to a 8 byte array.

Use case

This simple test consists of checking if a couple of values are indeed containing in the set. The filter will definitively reject 22 and very likely accept 5 & 97. If the objective is to confirm that 5 & 97 belong to the set, then a full-fledged hash table would have to be used.

val filter = new DigestBloomFilter[Long](100, 100)

val newValues = Array[Long](5L, 97L, 91L, 23L, 67L, 33L)
filter.addAll(newValues)

assert(filter.mightContain(5))
assert(filter.mightContain(97))
assert(!filter.mightContain(22))

Performance evaluation

Let's look at the behavior of the bloom filter under load. The test consists of adding 100,000,000 new random values then test if the filter contains a value (10,000) times. The test is run 10 times after a warm up of the JVM.

The first performance test evaluates the average time required to insert a new element into a Bloom filter which size range from 100M to 1Billion entries.
The second test evaluates the average search/query time for bloom filters with same range of size.

As expected the average time to load a new set of values and check the filter contains a specific value is fairly constant.

Spark-based filter

Apache Spark

Apache Spark is a free, open-source framework for cluster computing, specifically designed to process data in real time via distributed computing [ref 4]. Its primary applications include:

• Analytics: Spark's capability to quickly produce responses allows for interactive data handling, rather than relying solely on predefined queries.
• Data Integration: Often, the data from various systems is inconsistent and cannot be combined for analysis directly. To obtain consistent data, processes like Extract, Transform, and Load (ETL) are employed. Spark streamlines this ETL process, making it more cost-effective and time-efficient.
• Streaming: Managing real-time data, such as log files, is challenging. Spark excels in processing these data streams and can identify and block potentially fraudulent activities.
• Machine Learning: The growing volume of data has made machine learning techniques more viable and accurate. Spark's ability to store data in memory and execute repeated queries swiftly facilitates the use of machine learning algorithms.

Implementation

Apache Spark includes a Bloom filter implementation, BloomFilter, that's suitable for handling large datasets within data frames [ref 5]. It features two primary attributes:

The class SparkBloomFilter is parameterized with the type of elements T to be inserted and searched.

import org.apache.spark.util.sketch._

class SparkBloomFilter[T](bloomFilter: BloomFilter)  extends TBloomFilter[T] {

  
  def getExpectedFPRate: Double = bloomFilter.expectedFpp()

  override def mightContain(t: T): Boolean = bloomFilter.mightContain(t)

  override def add(t: T): Unit = bloomFilter.put(t)

  override def addAll(ts: Array[T]): Unit =
    if(ts.nonEmpty)
      ts.foreach(add)
}

The 3 methods are:

• add: Insert a new element into the filter
• addAll: Insert a set of elements into the filter
• mightContain: Test is the filter may contain a given item, with a  degree of certainty associated with the rate of false positives.

A generic constructor allows the customization of the Bloom filter on Spark with the following attributes:

• capacity: This refers to the maximum number of items that can be accommodated, as determined by hash functions.
• targetFPRate: This denotes the anticipated false positive rate, which is the frequency at which the filter incorrectly identifies an item as present in the set.

def apply[T](capacity: Int, targetFPRate: Float): SparkBloomFilter[T] =
    new SparkBloomFilter[T]( BloomFilter.create(capacity, targetFPRate))

Impact of capacity on expected false positives

The anticipated false-positive rate indicates the probability of hash function collisions. As the filter's capacity increases, it becomes less probable that a new item will clash with one already in the filter. We'll delve into how the filter's capacity influences the collision rate within the hashing mechanism.

def computeExpectedFPRate(capacity: Int, input: Array[Long]): Double = {
   val filter = SparkBloomFilter[Long](capacity, 0.05F)
  

   input.foreach(n => filter.add(n)) /
   filter.getExpectedFPRate
}


val input = Array[Long](5L, 97L, 91L, 23L, 67L, 33L) ++
      Array.tabulate(10000)(n => n.toLong+100L)
 
(1000 until 12000 by 500).foreach(
    capacity => println(s"$capacity ${computeExpectedFPRate(capacity, input)}")
)

In this experiment, we chose capacity values within the range of 1,000 to 12,000 for a Bloom filter containing 10,006 entries. The expected false-positive rate decreases from 1.0 towards nearly 0.0. Specifically, at a capacity of 10,006, where each entry is allocated a distinct slot, the observed rate of false positives at 0.0509 aligns with the predetermined target of 0.5.

Application to datasets

A Bloom filter can be utilized on a specific column, referred to as columnName, within a dataset containing a vast number of values, named dataSet. This application requires setting a defined capacity and aiming for a certain false positive rate.

def apply[T](
  dataSet: Dataset[T],
  columnName: String,
  capacity: Int,
  targetFPRate: Double)(implicit sparkSession: SparkSession): SparkBloomFilter[T]= {
    val filter = dataSet.stat.bloomFilter(columnName, capacity, targetFPRate)
    new SparkBloomFilter[T](filter)
}

Finally, we can apply this constructor to create a Bloom filter on a very large Spark Dataset.

case class TEntry(id: String, value: Float)

val dataSize = 1000000
val dataSet = Seq.tabulate(dataSize)(
    n => TEntry(n.toString, Random.nextFloat())
).toDS()

val filter = SparkBloomFilter(dataSet, "id", dataSize, 0.05)

Thank you for reading this article. For more information ...

References

[1] Bloom filter Wikipedia

[2] Geeks for Geeks: Message Digests

[3] Java API: Message Digest

[4] Apache Spark 3.4.0
[5] Spark BloomFilter

---------------------------

Patrick Nicolas has over 25 years of experience in software and data engineering, architecture design and end-to-end deployment and support with extensive knowledge in machine learning. 
He has been director of data engineering at Aideo Technologies since 2017 and he is the author of "Scala for Machine Learning" Packt Publishing ISBN 978-1-78712-238-3

Automate GAN Configuration in PyTorch

Target audience: Advanced
Estimated reading time: 5'

Posts history

Working with the architecture and deployment of Generative Adversarial Networks (GANs) often involves complex details that can be difficult to understand and resolve. Consider the advantage of identifying neural components that are reusable and can be utilized in both the generator and discriminator parts of the network.

Table of contents

The challenge

Convolutional networks

     Neural blocks

     Inverting a convolutional neural block

     Sizing de-convolutional layers

     Assembling the de-convolutional network

References

Notes:

• This post steers clear of the intricate technicalities of generative adversarial networks and convolutional neural networks. Instead, it focuses on automating the setup process for certain neural components.
• Readers are expected to have a foundational knowledge of neural networks and familiarity with the PyTorch library.
• Environments: Python 3.9, PyTorch 1.9.1

The challenge

This article is focused on streamlining the development of Deep Convolutional Generative Adversarial Networks (DCGANs) [ref 1]. We achieve this by configuring the generator in relation to the setup of the discriminator. The main area of our study is the well-known application of using GANs to differentiate between real and fake images.

For those unfamiliar with GANs..... 

Generative Adversarial Networks (GANs) [ref 2] are a type of unsupervised learning model that identify patterns within data and utilize these patterns for data augmentation, creating new samples that closely resemble the original dataset. GANs belong to the family of generative models, which also includes variational auto-encoders and maximum likelihood estimation (MLE) models. The unique aspect of GANs is that they convert the problem into a form of supervised learning by employing two competing networks:

• The Generator model, which is trained to produce new data samples.
• The Discriminator model, which aims to differentiate between real samples (from the original dataset) and fake ones (created by the Generator).

Crafting and setting up components like the generator and discriminator in a Generative Adversarial Network (GAN), or the encoder and decoder layers in a Variational Convolutional Auto-Encoder (VAE), can often be a repetitive and laborious process.

In fact, some aspects of this process can be entirely automated. For instance, the generative network in a convolutional GAN can be designed as the inverse of the discriminator using a de-convolutional network. Similarly, the decoder in a VAE can be automatically configured based on the structure of its encoder.

Functional representation of a simple deep convolutional GAN

Neural component reusability is key to generate a de-convolutional network from a convolutional network. To this purpose we break down a neural network into computational blocks.

Convolutional networks

In its most basic form, a Generative Adversarial Network (GAN) consists of two distinct neural networks: a generator and a discriminator.

Neural blocks

Each of these networks is further subdivided into neural blocks or groups of PyTorch modules, which include elements like hidden layers, batch normalization, regularization, pooling modes, and activation functions. Take, for example, a discriminator that is structured using a convolutional neural network [ref 3] followed by a fully connected (restricted Boltzmann machine) network. The PyTorch modules corresponding to each layer are organized into what we call a neural block class.

A PyTorch modules of the convolutional neural block [ref 4] are:

• Conv2d: Convolutional layer with input, output channels, kernel, stride and padding
• Dropout: Drop-out regularization layer
• BatchNorm2d: Batch normalization module
• MaxPool2d Pooling layer
• ReLu, Sigmoid, ... Activation functions

Representation of a convolutional neural block with PyTorch modules

The constructor of the neural block is designed to initialize all its parameters and modules in the correct sequence. For simplicity, we are not including regularization elements like drop-out (which essentially involves bagging of sub-networks) in this setup.

Important note: Each step of the algorithm makes reference to comments in the code (i.e.  The first step [1] is to initialize the number of input and output channels refers to  # [1] - initialize the input and output channels).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 3 4 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

class ConvNeuralBlock(nn.Module): def __init__(self, in_channels: int, out_channels: int, kernel_size: int, stride: int, padding: int, batch_norm: bool, max_pooling_kernel: int, activation: nn.Module, bias: bool, is_spectral: bool = False): super(ConvNeuralBlock, self).__init__() # Assertions are omitted # [1] - initialize the input and output channels self.in_channels = in_channels self.out_channels = out_channels self.is_spectral = is_spectral modules = [] # [2] - create a 2 dimension convolution layer conv_module = nn.Conv2d( self.in_channels, self.out_channels, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias) # [6] - if this is a spectral norm block if self.is_spectral: conv_module = nn.utils.spectral_norm(conv_module) modules.append(conv_module) # [3] - Batch normalization if batch_norm: modules.append(nn.BatchNorm2d(self.out_channels)) # [4] - Activation function if activation is not None: modules.append(activation) # [5] - Pooling module if max_pooling_kernel > 0: modules.append(nn.MaxPool2d(max_pooling_kernel)) self.modules = tuple(modules)

We considering the case of a generative model for images. The first step [#1] is to initialize the number of input and output channels, then create the 2-dimension convolution [#2], a batch normalization module [#3] an activation function [#4] and finally a max pooling module [#5]. The spectral norm regularization term [#6] is optional.
The convolutional neural network is assembled from convolutional and feedback forward neural blocks, in the following build method.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55

class ConvModel(NeuralModel): def __init__(self, model_id: str, # [1] - Number of input and output unites input_size: int, output_size: int, # [2] - PyTorch convolutional modules conv_model: nn.Sequential, dff_model_input_size: int = -1, # [3] - PyTorch fully connected dff_model: nn.Sequential = None): super(ConvModel, self).__init__(model_id) self.input_size = input_size self.output_size = output_size self.conv_model = conv_model self.dff_model_input_size = dff_model_input_size self.dff_model = dff_model @classmethod def build(cls, model_id: str, conv_neural_blocks: list, dff_neural_blocks: list) -> NeuralModel: # [4] - Initialize the input and output size # for the convolutional layer input_size = conv_neural_blocks[0].in_channels output_size = conv_neural_blocks[len(conv_neural_blocks) - 1].out_channels # [5] - Generate the model from the sequence # of conv. neural blocks conv_modules = [conv_module for conv_block in conv_neural_blocks for conv_module in conv_block.modules] conv_model = nn.Sequential(*conv_modules) # [6] - If a fully connected RBM is included in the model .. if dff_neural_blocks is not None and not is_vae: dff_modules = [dff_module for dff_block in dff_neural_blocks for dff_module in dff_block.modules] dff_model_input_size = dff_neural_blocks[0].output_size dff_model = nn.Sequential(*tuple(dff_modules)) else: dff_model_input_size = -1 dff_model = None return cls( model_id, conv_dimension, input_size, output_size, conv_model, dff_model_input_size, dff_model)

The standard constructor [#1] sets up the count of input/output channels, along with the PyTorch modules for the convolutional layers [#2] and the fully connected layers [#3].

The class method, build, creates the convolutional model using convolutional neural blocks and feed-forward neural blocks. It determines the dimensions of the input and output layers based on the first and last neural blocks [#4], and then produces the PyTorch convolutional modules [#5] and modules for fully-connected layers [#6] from these neural blocks.

Following this, we proceed to construct the de-convolutional neural network utilizing the convolutional blocks.

Inverting a convolutional block

To build a GAN, one must follow these steps:

1. Select and specify the PyTorch modules that will constitute each convolutional layer.
2. Assemble these chosen modules into a single convolutional neural block.
3. Construct the generator and discriminator of the GAN by integrating these neural blocks.
4. Link the generator and discriminator to create a fully functional GAN.

The aim here is to create a builder capable of producing the de-convolutional network. This network will act as the GAN's generator, formulated on the basis of the convolutional network described in the preceding section.

The process begins with the extraction of the de-convolutional block from an already established convolutional block.

Conceptual automated generation of de-convolutional block

The standard constructor for the neural block in a de-convolutional network sets up all the essential parameters required for the network, with the exception of the pooling module (which is not necessary). The code example provided demonstrates how to create a De-convolutional neural block. This process involves using convolution parameters like the number of input and output channels, kernel size, stride, padding, along with batch normalization and the activation function.

class DeConvNeuralBlock(nn.Module): def __init__(self, in_channels: int, out_channels: int, kernel_size: int, stride: int, padding: int, batch_norm: bool, activation: nn.Module, bias: bool) -> object: super(DeConvNeuralBlock, self).__init__() self.in_channels = in_channels self.out_channels = out_channels modules = [] # Two dimension de-convolution layer de_conv = nn.ConvTranspose2d( self.in_channels, self.out_channels, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias) # Add the deconvolution block modules.append(de_conv) # Add the batch normalization, if defined if batch_norm: modules.append(nn.BatchNorm2d(self.out_channels)) # Add activation modules.append(activation) self.modules = modules

Be aware that the de-convolution block lacks pooling capabilities. The class method named auto_build accepts a convolutional neural block, the number of input and output channels, and an optional activation function to create a de-convolutional neural block of the DeConvNeuralBlock type. The calculation of the number of input and output channels for the resulting deconvolution layer is handled by the private method __resize.

@classmethod def auto_build(cls, conv_block: ConvNeuralBlock, in_channels: int, out_channels: int = None, activation: nn.Module = None) -> nn.Module: # Extract the parameters of the source convolutional block kernel_size, stride, padding, batch_norm, activation = \ DeConvNeuralBlock.__resize(conv_block, activation) # Override the number of input_tensor channels # for this block if defined next_block_in_channels = in_channels if in_channels is not None \ else conv_block.out_channels # Override the number of output-channels for # this block if specified next_block_out_channels = out_channels if out_channels is not None \ else conv_block.in_channels return cls( conv_block.conv_dimension, next_block_in_channels, next_block_out_channels, kernel_size, stride, padding, batch_norm, activation, False)

Sizing de-convolutional layers

The next task consists of computing the size of the component of the de-convolutional block from the original convolutional block. 

@staticmethod
def __resize(
  conv_block: ConvNeuralBlock,
  updated_activation: nn.Module) -> (int, int, int, bool, nn.Module):
  conv_modules = list(conv_block.modules)
    
  # [1] - Extract the various components of the 
  #        convolutional neural block
  _, batch_norm, activation = DeConvNeuralBlock.__de_conv_modules(conv_modules)
  

  # [2] - override the activation function for the 
  #        output layer, if necessary
  if updated_activation is not None:
     activation = updated_activation
    
    # [3]- Compute the parameters for the de-convolutional 
    #       layer, from the conv. block
     kernel_size, _ = conv_modules[0].kernel_size
     stride, _ = conv_modules[0].stride
     padding = conv_modules[0].padding

 return kernel_size, stride, padding, batch_norm, activation

The __resize method performs several functions: it retrieves the PyTorch modules for the de-convolutional layers from the initial convolutional block [#1], incorporates the activation function into the block [#2], and ultimately sets up the parameters for the de-convolutional layer [#3].

Additionally, there's a utility method named __de_conf_modules. This method is responsible for extracting the PyTorch modules associated with the convolutional layer, the batch normalization module, and the activation function for the de-convolution, all from the convolution's PyTorch modules.

@staticmethod
def __de_conv_modules(conv_modules: list) -> \
        (torch.nn.Module, torch.nn.Module, torch.nn.Module):

  activation_function = None
  deconv_layer = None
  batch_norm_module = None

  # 4- Extract the PyTorch de-convolutional modules 
  #     from the convolutional ones
  for conv_module in conv_modules:
     if DeConvNeuralBlock.__is_conv(conv_module):
         deconv_layer = conv_module
     elif DeConvNeuralBlock.__is_batch_norm(conv_module):
         batch_norm_moduled = conv_module
     elif DeConvNeuralBlock.__is_activation(conv_module):
        activation_function = conv_module

  return deconv_layer, batch_norm_module, activation_function

Convolutional layers

and the height of the two dimension output data is

De-convolutional layers

As expected, the formula to compute the size of the output of a de-convolutional layer is the mirror image of the formula for the output size of the convolutional layer.

and

Assembling de-convolutional network

Finally, a de-convolutional model, categorized as DeConvModel, is constructed using a sequence of PyTorch modules, referred to as de_conv_model. The default constructor [#1] is used once more to establish the dimensions of the input layer [#2] and the output layer [#3]. It also loads the PyTorch modules, named de_conv_modules, for all the de-convolutional layers.

class DeConvModel(NeuralModel, ConvSizeParams):

  def __init__(self,            # [1] - Default constructor
           model_id: str,
           input_size: int,      # [2] - Size first layer
           output_size: int,    # [3] - Size output layer
           de_conv_modules: torch.nn.Sequential):
    super(DeConvModel, self).__init__(model_id)
    self.input_size = input_size
    self.output_size = output_size
    self.de_conv_modules = de_conv_modules


  @classmethod
  def build(cls,
      model_id: str,
      conv_neural_blocks: list,  # [4] - Input to the builder
      in_channels: int,
      out_channels: int = None,
      last_block_activation: torch.nn.Module = None) -> NeuralModel:
    
    de_conv_neural_blocks = []

    # [5] - Need to reverse the order of convolutional neural blocks
    list.reverse(conv_neural_blocks)

    # [6] - Traverse the list of convolutional neural blocks
    for idx in range(len(conv_neural_blocks)):
       conv_neural_block = conv_neural_blocks[idx]
       new_in_channels = None
       activation = None
       last_out_channels = None

        # [7] - Update num. input channels for the first 
        # de-convolutional layer
       if idx == 0:
            new_in_channels = in_channels
        
        # [8] - Defined, if necessary the activation 
        # function for the last layer
       elif idx == len(conv_neural_blocks) - 1:
          if last_block_activation is not None:
             activation = last_block_activation
          if out_channels is not None:
             last_out_channels = out_channels

        # [9] - Apply transposition to the convolutional block
      de_conv_neural_block = DeConvNeuralBlock.auto_build(
           conv_neural_block,
           new_in_channels,
           last_out_channels,
            activation)
      de_conv_neural_blocks.append(de_conv_neural_block)
        
       # [10]- Instantiate the Deconvolutional network 
       # from its neural blocks
   de_conv_model = DeConvModel.assemble(
       model_id, 
       de_conv_neural_blocks)
     
   del de_conv_neural_blocks
   return de_conv_model

The alternative constructor named build is designed to generate and set up the de-convolutional model using the convolutional blocks, referred to as conv_neural_blocks [#4].

To align the order of de-convolutional layers correctly, it's necessary to reverse the sequence of convolutional blocks [#5]. For every block within the convolutional network [#6], this method adjusts the number of input channels to match the number of input channels in the first layer [#7].

It then updates the activation function for the final output layer [#8] and systematically integrates the de-convolutional blocks [#9]. Ultimately, the de-convolutional neural network is composed using these blocks [#10]..

@classmethod
def assemble(cls, model_id: str, de_conv_neural_blocks: list):
   input_size = de_conv_neural_blocks[0].in_channels
   output_size = de_conv_neural_blocks[len(de_conv_neural_blocks)-1].out_channels 

   # [11]- Generate the PyTorch convolutional modules used by the default constructor
  conv_modules = tuple([conv_module for conv_block in de_conv_neural_blocks
                        for conv_module in conv_block.modules 
                        if conv_module is not None])
  de_conv_model = torch.nn.Sequential(*conv_modules)

  return cls(model_id, input_size, output_size, de_conv_model)

The assemble method is responsible for building the complete de-convolutional neural network. It does this by compiling the PyTorch modules from each of the blocks in de_conv_neural_blocks into a cohesive unit [#11].

Thank you for reading this article. For more information ...

References
[1] Tutorial: DCGAN in PyTorch
[2] A Gentle Introduction to Generative Adversarial Networks
[3] Deep learning Chap 9 Convolutional networks. 
          I. Goodfellow, Y. Bengio, A. Courville - 2017 MIT Press Cambridge MA 
[4] PyTorch
Source code: GitHub Neural Architecture


---------------------------
Patrick Nicolas has over 25 years of experience in software and data engineering, architecture design and end-to-end deployment and support with extensive knowledge in machine learning. 
He has been director of data engineering at Aideo Technologies since 2017 and he is the author of "Scala for Machine Learning" Packt Publishing ISBN 978-1-78712-238-3