Monday, August 19, 2024

Performance Improvement in Numpy 2.x & Lapack 3.9

Target audience: Beginner
Estimated reading time: 4'
The performance of training geometric learning and complex deep learning models directly affects time to market and development costs. Numpy 2.x, combined with the accelerator and the latest BLAS/LAPACK libraries, can reduce execution time by up to 38%.


Table of contents
Overview
Linear Algebra
      Evaluation
      Implementation
       Evaluation
Follow me on LinkedIn

What you will learn: Enhanced performance in linear algebra computations using Numpy 2.x with the ILP64 accelerator and LAPACK 3.9.1 library

Notes: 

  • Environments: Python  3.11,  Matplotlib 3.9, Numpy 1.2.6/2.1, LAPACK 3.9.1
  • Source code is available at Github.com/patnicolas/Data_Exploration/numpy2
  • To enhance the readability of the algorithm implementations, we have omitted non-essential code elements like error checking, comments, exceptions, validation of class and method arguments, scoping qualifiers, and import statements.


Overview

Non-linear deep learning and geometric learning depend on geodesics, Riemannian metrics, and exponential maps to train models and predict outcomes. These operations are computationally intensive, making them ideal candidates for performance enhancements in the latest release of Numpy, version 2.1 [ref 1]. 
Linear algebra operations, on the other hand, rely on the BLAS and LAPACK libraries, which can be optimized for specific platforms [ref 2].

This study is performed on a MacOS M2 Maxlaptop, utilizing optimizations of various mathematical libraries like LAPACK and BLAS from the Accelerate framework [ref 3]. Specifically, we use the ILP64 interface with the LAPACK 3.9.1 library alongside Numpy 2.1 [ref 4].

In the following two sections, we will assess the performance enhancements in Numpy 2.1 when utilizing the ILP64 accelerator for the LINPACK library on macOS. The evaluation will cover two scenarios:
- Basic Linear Algebra operations
- Fast Fourier Transform

Linear Algebra

Let's compare the performance of Numpy 2.1 with the LINPACK accelerator 3.9.1 against Numpy 1.2.6 by performing the following computations:
  • Sigmoid function on a Numpy array of varying sizes
  • Summation of two 3D Numpy arrays
  • Multiplication of two 3D Numpy arrays
  • Dot product of two arrays.

Implementation

Let's start by creating keys for our linear algebra operators.

SIGMOID: AnyStr = 'sigmoid'
ADD: AnyStr = 'add'
MUL: AnyStr = 'mul'
DOT: AnyStr = 'dot'


We wraps the basic operations on Numpy array into the class LinearAlgebraEval with two constructors
  • Default, __init__ with input array and shape
  • build method to generate a random 3D array.
The four methods/operations—sigmoid, add, mul, and dot product—are straightforward and self-explanatory.

class LinearAlgebraEval(object):
    def __init__(self, x: np.array, shape: List[int]) -> None:
        self.x = x.reshape(shape)

    @classmethod
    def build(cls, size: int) -> Self:
        _x = np.random.uniform(0.0, 100.0, size)
        return cls(_x, [10, 100, -1])

    def sigmoid(self, a: float) -> np.array:
        return 1.0/(1.0+ np.exp(-self.x * a))

    def add(self, y: np.array) -> np.array:
        return self.x + y.reshape(self.x.shape)

    def mul(self, y: np.array) -> np.array:
        z = 2.0* self.x * y.reshape(self.x.shape)
        return z * z

    def dot(self, y: np.array) -> np.array:
        return np.dot(self.x.reshape(-1), y.reshape(-1))

The performance evaluation is conducted by an instance of the LinearAlgebraPerfEval class. The key method __call__ relies on a dictionary with operation as key and list of execution times for these operations.

class LinearAlgebraPerfEval(object):
    def __init__(self, sizes: List[int]) -> None:
        self.sizes = sizes

    def __call__(self) -> Dict[AnyStr, List[float]]:
        _perf = {}
        lin_algebra = LinearAlgebraEval.build(self.sizes[0])
        _perf[SIGMOID] = [LinearAlgebraPerfEval.__performance(lin_algebra, SIGMOID)]
        _perf[ADD] = [LinearAlgebraPerfEval.__performance(lin_algebra, ADD)]
        _perf[MUL] = [LinearAlgebraPerfEval.__performance(lin_algebra, MUL)]
        _perf[DOT] = [ LinearAlgebraPerfEval.__performance(lin_algebra, DOT)]

        for index in range(1, len(self.sizes)):
            # Alternative constructor
            lin_algebra = LinearAlgebraEval.build(self.sizes[index])

            # The method __record executes each of the four operations for
            # a variable size of Numpy array.

            _perf = LinearAlgebraPerfEval.__record(_perf, lin_algebra, SIGMOID)
            _perf = LinearAlgebraPerfEval.__record(_perf, lin_algebra, ADD)
            _perf = LinearAlgebraPerfEval.__record(_perf, lin_algebra, MUL)
            _perf = LinearAlgebraPerfEval.__record(_perf, lin_algebra, DOT)
        return _perf

The private, helper method, __record described in the Appendix, executes and time  each of the four operations for set of Numpy arrays of increasing size. The timing relies on the timeit decorator described in the Appendix.

Evaluation

Let's plot the execution time for these 4 operations with a size of 3D arrays (10, 100, -1) varying between 1million to 60 million values.

Numpy 1.2.6 with OpenBlas


Numpy 2.1 with ILP64- BLAS/LAPACK 3.9.1



Numpy 2.1 and ILP-64 have no impact on the performance of the Sigmoid but reduce the average time to execute and addition, multiplication and dot product on these large arrays by 25 to 40%


Fast Fourier Transform

Let's define a signal as a sum of 4 sinusoidal functions with 4 different frequency modes.\[ f(t)=\sum_{i=1}^{4} sin(2.\pi.f_{i}.t)\]The signal is then sampled in the interval [0, 1] with various number of data points.

Implementation

We encapsulate the generation, sampling, and extraction of the frequency spectrum within the `FFTEval` class. 
The constructor iteratively adds sine functions to implement the desired formula. 
The `compute` method samples the signal over the interval [0, 1] and calls the Numpy `fft.fftfreq` method to generate the frequency distribution. The timing of the execution in the method compute uses the timeit decorator.

class FFTEval(object):
    def __init__(self, frequencies: List[int], samples: int) -> None:
        self.F = 1.0/samples
        self.x = np.arange(0, 1.0, self.F)
        pi_2 = 2*np.pi
        
       # Composed the sinusoidal signal/function
        self.signal = np.sin(pi_2*frequencies[0]*self.x)
        if len(frequencies) > 1:
            for f in frequencies[1:]:
                self.signal += np.sin(pi_2*f*self.x)

    @timeit
    def compute(self) -> (np.array, np.array):
        num_samples = len(self.signal)

        # Select the positive amplitudes and frequencies
        num_half_samples = num_samples//2

        # Invoke the Numpy FFT function to extract frequencies
        freqs = np.fft.fftfreq(num_samples, self.F)
        _freqs = freqs[:num_half_samples]
        _amplitudes = np.abs(np.fft.fft(self.signal)[:num_half_samples]) / num_samples
        return _freqs, _amplitudes

The class FFTPerfEval implements the execution and collection of the execution time for the FFT with various frequency of samples (number of samples in the interval [0, 1])

class FFTPerfEval(PerfEval):
    def __init__(self, sizes: List[int]) -> None:
        super(FFTPerfEval, self).__init__(sizes)

    def __call__(self) -> List[float]:
        durations = []

        # Collect the execution time for each of the number of
        # samples defined in the constructor
        for samples in self.sizes:
            durations.append(FFTPerfEval.__compute(samples))
        return durations

    @staticmethod
    def __compute(sz: int) -> float:
        frequencies = [4, 7, 11, 17]
        fft_eval = FFTEval(frequencies, sz)
        # Time it
        return fft_eval.compute()


Evaluation

Numpy 1.2.6 with OpenBlas



Numpy 2.1 with BLAS/LAPACK 3.9.1



Relative performance improvement


The switch to Numpy 2.1 with ILP-64 on MacOS 14.6.1 shows an average improvement of 33%

References

[1] Numpy
[2] Lapack


------------------
Patrick Nicolas has over 25 years of experience in software and data engineering, architecture design and end-to-end deployment and support with extensive knowledge in machine learning. 
He has been director of data engineering at Aideo Technologies since 2017 and he is the author of "Scala for Machine Learning", Packt Publishing ISBN 978-1-78712-238-3 and Geometric Learning in Python Newsletter on LinkedIn.

Appendix

Standard timing decorator in Python

def timeit(func):
    import time

    def wrapper(*args, **kwargs):
        start_time = time.time()
        func(*args, **kwargs)
        end_time = time.time()
        return end_time - start_time
    return wrapper


Methods to compute and record the execution time for basic linear algebra operations. The execution time is recorded through the decorator timer

 @staticmethod
    def __record(perf: Dict[AnyStr, List[float]],
                 lin_algebra: LinearAlgebraEval,
                 op: AnyStr) -> Dict[AnyStr, List[float]]:
        duration = LinearAlgebraPerfEval.__performance(lin_algebra, op)
        lst = perf[op]
        lst.append(duration)
        perf[op] = lst
        return perf

    @timeit
    @staticmethod
    def __performance(lin_algebra: LinearAlgebraEval, op: AnyStr) -> np.array:
        match op:
            case "sigmoid":
                return lin_algebra.sigmoid(8.5)
            case "add":
                return lin_algebra.add(lin_algebra.x)
            case "mul":
                return lin_algebra.mul(lin_algebra.x)
            case "dot":
                return lin_algebra.dot(lin_algebra.


Posted by
Labels: , , , , , , ,

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Subscribe to: Post Comments (Atom)