Target audience: Advanced

Estimated reading time: 6'

As a machine learning engineer, I've found it challenging to encounter the same level of reusability and design patterns in this field that are common in conventional software development. Implementations of deep learning models often depend heavily on repetitive, boilerplate code.

In this post, we explore the idea of reusable neural blocks, a straightforward and practical approach for packaging and reusing components of neural networks. Specifically, we'll delve into creating neural blocks for a variational auto-encoder using PyTorch, as well as for a Bidirectional Embeddings Representation from Transformers (BERT) encoder, utilizing the Deep Java Library.

Table of contents

Reusable neural blocks

Neural blocks in PyTorch

Modular convolutional neural networks

Modular variational auto-encoders

Neural blocks in Deep Java Library

References

Notes:

Source code is available on GitHub Github Neural Architecture
Environments: Python 3.10, PyTorch 2.1.1
To enhance the readability of the algorithm implementations, we have omitted non-essential code elements like error checking, comments, exceptions, validation of class and method arguments, scoping qualifiers, and import statements.

Reusable neural blocks

Complex deep learning models have large stack of neural transformation such as convolution, fully connected network layers, activations, regularization modules, loss functions or embedding layers.

Creating these models using basic components from existing deep learning library is a daunting task. A neural block aggregates multiple components of a neural network into a logical, clearly defined function or task. A block is a transformation in the data flow used in training and inference.

Neural blocks in PyTorch

Modular convolutional neural network

A convolutional neural network can be broken down into neural blocks that organize PyTorch modules such as hidden layers, input and output channels, batch normalization, regularization, pooling mode and activation function into a single computation unit.

First, let's consider a conventional convolutional neural network with a fully connected (restricted Boltzmann machine) network. The PyTorch modules associated with any given layer are assembled as a neural block class.

A PyTorch modules of the convolutional neural block are:

Conv2d: Convolutional layer with input, output channels, kernel, stride and padding
Dropout: Drop-out regularization layer
BatchNorm2d: Batch normalization module
MaxPool2d Pooling layer
ReLu, Sigmoid, ... Activation functions

Here is a schematic representation of a convolutional neural network as a stack of neural blocks.

The constructor for the neural block class initializes all its parameters and its modules in the proper order. For the sake of simplicity, regularization elements such as drop-out (bagging of sub-network) is omitted.

class ConvNeuralBlock(nn.Module):
  def __init__(self,
      in_channels: int,
      out_channels: int,
      kernel_size: int,
      stride: int,
      padding: int,
      batch_norm: bool,
      max_pooling_kernel: int,
      activation: nn.Module,
      bias: bool,
      is_spectral: bool = False):
    
   super(ConvNeuralBlock, self).__init__()
        
   # Assertions are omitted
   # 1- initialize the input and output channels
   self.in_channels = in_channels
   self.out_channels = out_channels
   self.is_spectral = is_spectral
   modules = []
   
   # 2- create a 2 dimension convolution layer
   conv_module = nn.Conv2d(   
       self.in_channels,
       self.out_channels,
       kernel_size=kernel_size,
       stride=stride,
       padding=padding,
       bias=bias)

   # 6- if this is a spectral norm block
   if self.is_spectral:        
     conv_module = nn.utils.spectral_norm(conv_module)
     modules.append(conv_module)
        
   # 3- Batch normalization
   if batch_norm:               
     modules.append(nn.BatchNorm2d(self.out_channels))
     
   # 4- Activation function
   if activation is not None: 
     modules.append(activation)
        
   # 5- Pooling module
   if max_pooling_kernel > 0:   
     modules.append(nn.MaxPool2d(max_pooling_kernel))
   
   self.modules = tuple(modules)

The code snippet describes the various stages of building a convolutional block. The first step (1) is to initialize the number of input and output channels, then create the 2-dimension convolution (2), a batch normalization module (3) an activation function (4) and finally a Max pooling module (5). The spectral norm regularization (6) is optional.

Next, we package the various convolutional and feedback forward neural blocks
into a full fledge convolutional model, in the following build method.

class ConvModel(NeuralModel):
  def __init__(self,                    
       model_id: str,
       # 1 Number of input and output unites
       input_size: int,
       output_size: int,
       # 2- PyTorch convolutional modules
       conv_model: nn.Sequential,
       dff_model_input_size: int = -1,
       # 3- PyTorch fully connected
       dff_model: nn.Sequential = None):
        
   super(ConvModel, self).__init__(model_id)
   self.input_size = input_size
   self.output_size = output_size
   self.conv_model = conv_model
   self.dff_model_input_size = dff_model_input_size
   self.dff_model = dff_model
   
  @classmethod
  def build(cls,
      model_id: str,
      conv_neural_blocks: list,  
      dff_neural_blocks: list) -> NeuralModel:
            
   # 4- Initialize the input and output size 
   # for the convolutional layer
   input_size = conv_neural_blocks[0].in_channels
   output_size = conv_neural_blocks[len(conv_neural_blocks) - 1].out_channels

   # 5- Generate the model from the sequence 
   # of conv. neural blocks
   conv_modules = [conv_module for conv_block in conv_neural_blocks
         for conv_module in conv_block.modules]
   conv_model = nn.Sequential(*conv_modules)

   # 6- If a fully connected RBM is included in the model ..
   if dff_neural_blocks is not None and not is_vae:
     dff_modules = [dff_module for dff_block in dff_neural_blocks
        for dff_module in dff_block.modules]
         
     dff_model_input_size = dff_neural_blocks[0].output_size
     dff_model = nn.Sequential(*tuple(dff_modules))
   else:
     dff_model_input_size = -1
     dff_model = None
      
  return cls(
     model_id, 
     conv_dimension, 
     input_size, 
     output_size, 
     conv_model,
     dff_model_input_size, 
     dff_model)

The default constructor (1) initializes the number of input/output channels, the PyTorch modules for the convolutional layers (2) and the fully connected layers (3).
The class method, build, instantiates the convolutional model from several convolutional neural blocks and one feed forward neural block. It initializes the size of input and output layers from the first and last neural blocks (4), generate the PyTorch convolutional modules (5) and fully-connected layers' modules (6) from the neural blocks.

Modular variational auto-encoder

A de-convolutional neural network, DeConvModel is created from the convolutional model, ConvModel through reflection (see Automating the configuration of a GAN in PyTorch for more details). A mean, variance and sampling PyTorch modules are packaged into a variational neural block, VAENeuralBlock.

Finally, the variational auto-encoder, VAE is assembled by stacking the convolutional, variational and de-convolutional neural blocks.

Neural blocks in Deep Java Library

Deep Java Library (DJL) is an Apache open-source Java framework that supports the most commonly used deep learning frameworks; MXNet, PyTorch and TensorFlow. DJL ability to leverage any hardware configuration (CPU, GPU) and integrated with big data frameworks makes it and ideal solution for a highly performant distributed inference engine. DJL can be optionally used for training.

Everyone who has been involved with GPT-3 or GPT-4 decoder (ChatGPT) is aware of the complexity and interaction of neural components in transformers.

Let's apply DJL to build a BERT transformer encoder using neural blocks, knowing

A BERT encoder is a stack of multiple transformer modules
Pre-training block which contains BERT block, Masked Language Model (MLM) module and Next Sentence Predictor (NSP) with their associated loss functions
A BERT block is composed of embedding block.

The following Scala code snippet illustrates the composition of a BERT pre-training block using the transformer encoder block, thisTransformerBlock, the Masked Language Model component, thisMlmBlock and the Next Sentence Prediction module, thisNspBlock.

class CustomPretrainingBlock protected (
    mlmActivation: String
) extends AbstractBaseBlock {

  lazy val activationFunc: java.util.function.Function[NDArray, NDArray] =    
       ActivationConfig.getNDActivationFunc(activationType)

  // Transformer encoder block  // 1- Initialize the shape of tensors for the encoder, MLM and NSP blocks

  lazy val thisTransformerBlock: BertBlock = BertBlock.builder().base()
     .setTokenDictionarySize(Math.toIntExact(vocabularySize))
     .build

  // MLM block

  lazy val thisMlmBlock: BertMaskedLanguageModelBlock = 
       new BertMaskedLanguageModelBlock(bertBlock, activationFunc)

  // NSP block
  lazy val thisNspBlock: BertNextSentenceBlock =
       new BertNextSentenceBlock

  // 1- Initialize the shape of tensors for the encoder, MLM and NSP blocks

  override def initializeChildBlocks(
      ndManager: NDManager, 
      dataType: DataType, 
      shapes: Shape*): Unit

  
  // 2- Forward execution (i.e. PyTorch forward / __call__

  override protected def forwardInternal(
      parameterStore: ParameterStore,
      inputNDList: NDList,
      training : Boolean,
      params: PairList[String, java.lang.Object]): NDList 
}

DJL provides developers with two important methods

initializeChildBlock (1) initializes the shape of the tensors for the inner/child blocks
forwardInternal (2) implement the forward execution of neural network for the transformer and downstream classifier.

def forwardInternal(
    parameterStore: ParameterStore,
    inputNDList: NDList,
    training : Boolean,
    params: PairList[String, java.lang.Object]): NDList = {
 
    // Dimension batch_size x max_sentence_size
  val tokenIds = inputNDList.get(0)
  val typeIds = inputNDList.get(1)
  val inputMasks = inputNDList.get(2)


    // Dimension batch_size x num_masked_token
  val maskedIndices = inputNDList.get(3)

  val ndChildManager = NDManager.subManagerOf(tokenIds)
  ndChildManager.tempAttachAll(inputNDList)

      // Step 1: Process the transformer block for Bert
  val bertBlockNDInput = new NDList(tokenIds, typeIds, inputMasks)
  val ndBertResult = thisTransformerBlock.forward(
    parameterStore, 
    bertBlockNDInput, 
    training)

      // Step 2 Process the Next Sentence Predictor block
      // Embedding sequence dimensions are

      // batch_size x max_sentence_size x embedding_size
  val embeddedSequence = ndBertResult.get(0)
  val pooledOutput = ndBertResult.get(1)

      // Need to un-squeeze for batch size =1,

      // (embedding_vector) => (1, embedding_vector)
  val unSqueezePooledOutput =
     if(pooledOutput.getShape().dimension() == 1) {
       val expanded = pooledOutput.expandDims(0) 
       ndChildManager.tempAttachAll(expanded)
       expanded
     }
     else
      pooledOutput

      // We compute the NSP probabilities in case there are more than

      // a single sentence
  val logNSPProbabilities: NDArray =
       thisNspBlock.forward(
     parameterStore, 
     new NDList(unSqueezePooledOutput),
     training
  ).singletonOrThrow

        // Step 3: Process the Masked Language Model block
        // Embedding table dimension are vocabulary_size x Embeddings size
  val embeddingTable = thisTransformerBlock
            .getTokenEmbedding
            .getValue(parameterStore, embeddedSequence.getDevice(), training)

        // Dimension:  (batch_size x maskSize) x Vocabulary_size
  val logMLMProbabilities: NDArray = thisMlmBlock.forward(
      parameterStore,
      new NDList(embeddedSequence, maskedIndices, embeddingTable),
      training
  ).singletonOrThrow

        // Finally build the output
  val ndOutput = new NDList(logNSPProbabilities, logMLMProbabilities)
  ndChildManager.ret(ndOutput)

Thank you for reading this article. For more information ...

References

PyTorch
Deep Java Library
Automating the configuration of a GAN in PyTorch
github.com/patnicolas
Environments: Python 3.8, PyTorch 1.8.1, Scala 2.12.15, Java 11, Deep Java Library 0.20.0

---------------------------

Patrick Nicolas has over 25 years of experience in software and data engineering, architecture design and end-to-end deployment and support with extensive knowledge in machine learning.
He has been director of data engineering at Aideo Technologies since 2017 and he is the author of "Scala for Machine Learning" Packt Publishing ISBN 978-1-78712-238-3

Monday, November 21, 2022

Accelerate Deep Learning with Neural Blocks

Reusable neural blocks

Neural blocks in PyTorch

Modular convolutional neural network

Modular variational auto-encoder

Neural blocks in Deep Java Library

References

Contact Form

Equation Editor

Popular Posts