I delve into a diverse range of topics, spanning programming languages, machine learning, data engineering tools, and DevOps. Our articles are enriched with practical code examples, ensuring their applicability in real-world scenarios.
Showing posts with label Machine learning. Show all posts
Showing posts with label Machine learning. Show all posts
Friday, April 19, 2024
Thursday, March 14, 2024
Geometric Learning in Python: Differential Operators
Target audience: Intermediate
Estimated reading time: 5'
Estimated reading time: 5'
The use of Riemannian manifolds and topological computing is gaining traction in machine learning circles as a means to surpass the existing constraints of data analysis within Euclidean space.
This article aims to present the fundamental aspects of differential geometry.
Table of content
What you will learn: Vector fields, Differential operators and integral transforms as key components of differential geometry applied to deep learning.
Notes:
- Environments: Python 3.10.10 and SymPy 1.12
- The differential operators are used in a future post dedicated to differential geometry and manifold learning
- This article assumes that the reader is familiar with differential calculus and basic concept of geometry
- Source code is available at github.com/patnicolas/Data_Exploration/diffgeometry
- To enhance the readability of the algorithm implementations, we have omitted non-essential code elements like error checking, comments, exceptions, validation of class and method arguments, scoping qualifiers, and import statements.
Overview
The heavy dependence on elements from Euclidean space for data analysis and pattern recognition in machine learning might be approaching a threshold. Incorporating differential geometry and Riemannian manifolds has already shown beneficial effects on the accuracy of deep learning models, as well as on reducing their development costs [ref 1].
Why differential geometry
Conventional approaches in machine learning, pattern recognition, and data analysis often presuppose that input data can be effectively represented using elements from Euclidean space. Although this assumption has been sufficient for numerous applications in the past, there's a growing awareness among data scientists and engineers that most data in vision and pattern recognition actually reside in a differential manifold (feature space) that's embedded within the raw data's Euclidean space.
Leveraging this geometric information can result in a more precise representation of the data's inherent structure, leading to improved algorithms and enhanced performance in real-world applications [ref 2].
Why SymPy
SymPy is a Python library dedicated to symbolic mathematics. Its implementation is as simple as possible in order to be comprehensible and easily extensible. It is licensed under BSD. SymPy supports differential and integral calculus, Matrix operations, algebraic and polynomial equations, differential geometry, probability distributions, 3D plotting and discrete math [ref 3].
To install pip install sympy
Source code for SymPy is available at github.com/sympy/sympy.git
This article has 3 sections:
- Introduction of vector fields and implementation in SymPy
- Differential operators (Gradient, Divergence and Curl)
- Laplace and Fourier Transform
Vector fields
Let's start with an overview of vector fields as the basis of differential geometry [ref 4].
Consider a three-dimensional Euclidean space characterized by basis vectors {ei} and a vector field V, expressed as (f1, f2, f3). In this scenario, it is crucial to note that we are following the conventions of Einstein's tensor notation.
Visualization of a vector field in 3D Euclidean space
The vector field at the point P(x,y,z) is defined as the tuple (f1(x,y,z), f2(x,y,z), f3(x,y,z)). The vector over a field of n dimension field and basis vectors {ei} can be formally defined as \[f: \boldsymbol{x} \,\, \epsilon \,\,\, \mathbb{R}^{n} \mapsto \mathbb{R} \\ f(\mathbf{x})=\sum_{i=1}^{n}{f^{i}}(\mathbf{x}).\mathbf{e}_{i}\] Example for 3 dimension space: \[f(x,y,z) = (2x+z^{3})\boldsymbol{\mathbf{\overrightarrow{i}}} + (xy+e^{-y}-z^{2})\boldsymbol{\mathbf{\overrightarrow{j}}} + \frac{x^{3}}{y}\boldsymbol{\mathbf{\overrightarrow{k}}}\] Let's carry out the creation of this vector field, f, in SymPy and compute its value at f(1.0, 2.0, 0.5). To begin, we need to establish a 3D coordinate system, denoted as r, and then express the vector using r.
from sympy.vector import CoordSys3Dfrom sympy import exp
r = CoordSys3D('r')
f = (2*r.x + r.z**3)*r.i + (r.x*r.y+sympy.exp(-r.y)-r.z**2)*r.j + (r.x**3/r.y)*r.k
w = f.evalf(subs={r.x: 1.0, r.y: 2.0, r.z: 0.2})
print(w) # 2.008*r.i + 2.09533528323661*r.j + 0.5*r.k
Now, let's consider the same vector V with a second reference; origin 0' and basis vector e'i
Visualization of two coordinate systems
\[f(\mathbf{x})=\sum_{i=1}^{n} f'^{i}(\mathbf{x}).\mathbf{e'}_{i}\] The transformation matrix Sij convert the coordinates value functions fi and f'i. The tuple f =(fi) is the co-vector field for the vector field V
\[S_{ij}: \begin{Vmatrix} f^{1} \\ f^{2} \\ f^{3} \end{Vmatrix} \mapsto \begin{Vmatrix} f'^{1} \\ f'^{2} \\ f'^{3} \end{Vmatrix}\] The scalar product of the co-vector f' and vector v(f)
Differential operators
Let's demonstrate how to calculate differential operators using the SymPy library within a 3D Cartesian coordinate system. To begin, we'll create a class named DiffOperators which will encapsulate the implementation of various differential operators and transforms.
Gradient
Consider a scalar field f
class DiffOperators(object):
def __init__(self, expr: Expr):
self.expr = expr
def gradient(self) -> VectorZero:
from sympy.vector import gradient
return gradient(self.expr, doit=True)Example of input function: \[f(x,y,z)=x^{2}yz\]
r = CoordSys3D('r')
this_expr = r.x*r.x*r.y*r.z
diff_operator = DiffOperators(this_expr)
grad_res = diff_operator.gradient()
\[\triangledown f(x,y,z) = 2xyz.\vec{i} + x^{2}z.\vec{j} + x^{2}y.\vec{k}\]
Divergence
Divergence is a vector operator used to quantify the strength of a vector field's source or sink at a specific point, producing a signed scalar value. When applied to a vector F, comprising components X, Y, and Z, the divergence operator consistently yields a scalar result [ref 6].
\[div(F)=\triangledown .F=\frac{\partial X}{\partial x}+\frac{\partial Y}{\partial y}+\frac{\partial Z}{\partial z}\]
def divergence(self, base_vec: Expr) -> VectorZero:
from sympy.vector import divergence
div_vec = self.expr*base_vec
return divergence(div_vec, doit=True)Using the same input expression as used for the gradient calculation:
div_res = diff_operator.divergence(r.i + r.j + r.k)
\[div(f(x,y,z)[\vec{i} + \vec{j}+ \vec{k}]) = 2x(y+z+xy)\]
Curl
In mathematics, the curl operator represents the minute rotational movement of a vector in three-dimensional space. This rotation's direction follows the right-hand rule (aligned with the axis of rotation), while its magnitude is defined by the extent of the rotation [ref 6]. Within a 3D Cartesian system, for a three-dimensional vector F, the curl operator is defined as follows:
\[ \triangledown * \mathbf{F}=\left (\frac{\partial F_{z}}{\partial y}- \frac{\partial F_{y}}{\partial z} \right ).\vec{i} + \left (\frac{\partial F_{x}}{\partial z}- \frac{\partial F_{z}}{\partial x} \right ).\vec{j} + \left (\frac{\partial F_{y}}{\partial x}- \frac{\partial F_{x}}{\partial y} \right ).\vec{k} \]
def curl(self, base_vectors: Expr) -> VectorZero:
from sympy.vector import curl
curl_vec = self.expr*base_vectors
return curl(curl_vec, doit=True)
If we use only the two base vectors j and k:
curl_res = diff_operator.curl(r.j+r.k)
\[curl(f(x,y,z)[\vec{j} + \vec{k}]) = x^{2}\left ( z-y \right ).\vec{i}-2xyz.\vec{i}+2xyz.\vec{k}\]
Validation
We can confirm the accuracy of the gradient, divergence, and curl operators implemented in SymPy by utilizing the subsequent formulas: \[curl(\triangledown f))= \triangledown *\left ( \triangledown f\right ) = 0\] and \[\triangledown . curl (F) = \triangledown .\left ( \triangledown * F \right ) = 0\]
grad_res = diff_operator.gradient()
assert(diff_operator.curl(grad_res) ==0) # Print 0
curl_res = diff_operator.curl(r.i + r.j + r.k)
assert(diff_operator.divergence(curl_res) == 0) # Print 0
Transforms
An integral transform transforms a function from the time domain into its equivalent representation in the frequency domain, as depicted in the following diagram:
The evaluation of SymPy functions follows a two-step process:
- Generate function F in frequency space
- Evaluate output of F for given input values.
Laplace
The Laplace transform is a type of integral transform that changes a function of a real variable (typically time, t) into a function in the s-plane, which is a variable in the complex frequency domain [ref 7].. Specifically, it transforms ordinary differential equations into algebraic ones and converts convolution operations into simpler multiplication. The formulation of the Laplace transform is:\[\mathfrak{L}[f(t)](s)=\int_{0}^{+\inf}f(t).e^{-st}dt\]
def laplace(self):
from sympy import laplace_transform
return laplace_transform(self.expr, t, s, noconds=True)
Let's create the Laplace transforms for two basic functions and then calculate the outcomes of these transformed functions in the frequency domain, using the values s=0.5 and a=0.2.
t, s = sympy.symbols('t, s', real=True)
a = sympy.symbols('a', real=True)
diff_operator = DiffOperators(sympy.exp(-a*t))
laplace_func = diff_operator.laplace()
print(Laplace_func.evalf(subs={s:0.5, a:0.2})) # 1.4285\[\int_{0}^{+\inf}e^{-at}.e^{-st}dt = \frac{1}{a+s}\]
diff_operator = DiffOperators(sympy.sqrt(-a*t))
laplace_func = diff_operator.laplace() print(Laplace_func.evalf(subs={s:0.5, a:0.2})) # 1.1209\[\int_{0}^{+\inf}\sqrt{at}.e^{-st}dt = \frac{\sqrt{a\pi}}{2.s^{3/2}}\]
Fourier
Like the Laplace transform, the Fourier transform is another integral transform that translates a function into a version that represents the frequencies contained in the original function. The result of this transformation is a complex-valued function dependent on frequency [ref 8]. Often referred to as the frequency domain representation of the original function, the formulation of the Fourier transform is as follows: \[F(x )=\int_{-\inf}^{+\inf}f(t).e^{-2\pi itx}dt\] The implementation uses the SymPy function fourier_transform which takes the function (self.expr) and x, and k parameters as arguments.
def fourier(self):
from sympy import fourier_transform
return fourier_transform(self.expr, x, k)
Let's implement the Fourier transforms for two simple functions, exp and sin, and then calculate the outcomes of these transformed functions in the frequency domain with k=0.4
x, k = sympy.symbols('x, k', real=True)
diff_operator = DiffOperators(sympy.exp(-x**2))
fourier_func = diff_operator.fourier()
print(fourier_func.evalf(subs={k: 0.4})) # 0.3654
\[\int_{-\inf}^{+\inf}e^{-t^{2}}.e^{-2\pi itx}dt = \sqrt{\pi}.e^{-2\pi x^{2}}\]
diff_operator = DiffOperators(sympy.sin(x))
diff_operator.fourier()
\[\int_{-\inf}^{+\inf}\sin(t).e^{-2\pi itx}dt = 0\]
Thank you for reading this article. For more information ...
References
---------------------------
Patrick Nicolas has over 25 years of experience in software and data engineering, architecture design and end-to-end deployment and support with extensive knowledge in machine learning.
He has been director of data engineering at Aideo Technologies since 2017 and he is the author of "Scala for Machine Learning" Packt Publishing ISBN 978-1-78712-238-3
He has been director of data engineering at Aideo Technologies since 2017 and he is the author of "Scala for Machine Learning" Packt Publishing ISBN 978-1-78712-238-3
Monday, October 9, 2023
Automate Medical Coding Using BERT
Target audience: Beginner
Estimated reading time: 5'
Estimated reading time: 5'
Transformers and self-attention models are increasingly taking center stage in the NLP toolkit of data scientists [ref 1]. This article delves into the design, deployment, and assessment of a specialized transformer tasked with extracting medical codes from Electronic Health Records (EHR) [ref 2]. The focus is on curbing development and training expenses while ensuring the model remains current.
Table of contents
Introduction Extracting medical codes
Medical coding is the transformation of healthcare diagnosis, procedures, medical services described in electronic health records, physician's notes or laboratory results into alphanumeric codes. This study focuses on automated generation of medical codes and health insurance claims from a given clinical note or electronic health record.
fig. 6 Simplified data pipeline for active learning.
Important notes:
- This piece doesn't serve as a primer or detailed account of transformer-based encoders, Bidirectional Encoder Representations from Transformers (BERT), multi-label classification or active learning. Detailed and technical information on these models is available in the References section. [ref 1, 3, 8, 12].
- The terms medical document, medical note and clinical notes are used interchangeably
- Some functionalities discussed here are protected intellectual property, hence the omission of source code.
Introduction
Autonomous medical coding refers to the use of artificial intelligence (AI) and machine learning (ML) technologies to automatically assign medical codes to patient records [ref 4]. Medical coding is the process of assigning standardized codes to diagnoses, medical procedures, and services provided during a patient's visit to a healthcare facility. These codes are used for billing, reimbursement, and research purposes.
By automating the medical coding process, healthcare organizations can improve efficiency, accuracy, and consistency, while also reducing costs associated with manual coding.
A health insurance claim is an indication of the service given by a provider, even though the medical records associated with this service can greatly vary in content and structure. It's crucial to precisely extract medical codes from clinical notes since outcomes, like hospitalizations, treatments, or procedures, are directly tied to these diagnostic codes. Even if there are minor variations in the codes, claims can still be valid for specific services, provided the clinical notes, patient history, diagnosis, and advised procedures align.
fig. 1 Extraction of knowledge, predictions from electronic medical records
Challenges
There are 3 issues to address:
- How to extract medical codes reliably, given that labeling of medical codes is error prone and the clinical documents are very inconsistent?
- How to minimize the cost of self- training complex deep models such as transformers while preserving an acceptable accuracy?
- How to continuously keep models up to date in production environment?
Extracting medical codes
Medical codes are derived from patient records and clinical notes to forecast procedural results, determine the length of hospital stays, or generate insurance claims. The most prevalent medical coding systems include:
- International Classification of Diseases (ICD-10) for diagnosis (with roughly 72,000 codes)
- Current Procedural Terminology (CPT) for procedures and medications (encompassing around 19,000 codes)
- Along with others like Modifiers, SNOMED, and so forth.
The vast array of medical codes poses significant challenges in extraction due to:
- The seemingly endless combinations of codes linked to a specific medical document
- Varied and inconsistent formats of patient records (in terms of terminology, structure, and length.
- Complications in gleaning context from medical information systems.
Minimizing costs
A study on deep learning models suggests that training a significant language model (LLM) results in the emission of 626,155 pounds of CO2, comparable to the total emissions from five vehicles over their lifespan.
To illustrate, GPT-3/ChatGPT underwent training on 500 billion words with a model size of 175 billion parameters. A single training session would require 355 GPU-years and bear a cost of no less than $4.6M. Efforts are currently being made to fine-tune resource utilization for the development of upcoming models [ref 5].
Keeping models up-to-date
Customer data in real-time is continuously changing, often deviating from the distribution patterns the models were originally trained on (due to concept and covariate shifts).
This challenge is particularly pronounced for transformers that need task-specific fine-tuning and might even necessitate restarting the pre-training process — both of which are resource-intensive actions.
Architecture
To tackle the challenges highlighted earlier, the proposed solution should encompass four essential AI/NLP elements:
- Tokenizer to extract tokens, segments & vocabulary from a corpus of medical documents.
- Bidirectional Encoder Representations from Transformers (BERT) to generate a representation (embedding) of the documents [ref 3].
- Neural-based classifier to predict a set of diagnostic codes or insurance claim given the embeddings.
- Active/transfer learning framework to update model through optimized selection/sampling of training data from production environment.
From a software engineering perspective, the system architecture should provide a modular integration capability with current IT infrastructures. It also requires an asynchronous messaging system with streaming capabilities, such as Kafka, and REST API endpoints to facilitate testing and seamless production deployment.
fig. 2 Architecture for integration of AI components with external medical IT systems
Tokenizer
The effectiveness of a transformer encoder's output hinges on the quality of its input: tokens and segments or sentences derived from clinical documents. Several pressing questions need addressing:
- Which vocabulary is most suitable for token extraction from these notes? Do we consider domain-specific terms, abbreviations, Tf-Idf scores, etc.?
- What's the best approach to segmenting a note into coherent units, such as sections or sentences?
- How do we incorporate or embed pertinent contextual data about the patient or provider into the encoder?
Tokens play a pivotal role in formulating a dynamic vocabulary. This vocabulary can be enriched by incorporating words or N-grams from various sources like:
- Terminology from the American Medical Association (AMA)
- Common medical terms with high TF-IDF scores
- Different senses of words
- Abbreviations
- Semantic descriptions
- Stems
- .....
fig. 3 Generation of a vocabulary using training corpus and knowledge base
Our optimal approach is based on utilizing uncased words from the American Medical Association, coupled with the top 85% of terms derived from training medical notes, ranked by their highest TF-IDF scores. It's worth noting that this method can be resource-intensive.
BERT encoder
In NLP, words and documents are represented in the form of numeric vectors allowing similar words to have similar vector representations [ref 6].
The objective is to generate embeddings for medical documents including contextual data to be feed into a deep learning classifier to extract diagnostic codes or generate a medical insurance claim [ref 7].
Context embedding
Contextual information such as patient data (age, gender,...), medical service provider, specialty, or location is categorized (or bucked for continuous values) and added to the tokens extracted from the medical note.
Segmentation
Structuring electronic health records into logical or random groups of segments/sentences presents a significant challenge. Segmentation involves dividing a medical document into segments (or sections), each with an equal number of tokens that consist of sentences and relevant contextual data.
Several methods can be employed to segment a document:
- Isolating the contextual data as a standalone segment.
- Integrating the contextual data into the document's initial segment.
- Embedding the contextual data into any arbitrarily chosen segment [Ref 6].
fig. 4 Embedding of medical note with contextual data using 2 segments
Our study show the option 2 provides the best embedding for the feed forward neural network classifier.
Interestingly, treating the entire note as a single sentence and using the AMA vocabulary leads to diminished accuracy in subsequent classification tasks.
Transformer
We employ the self-supervised Bidirectional Representation for Transformer (BERT) with the objectives to:
- Grasp the contextual significance of medical phrases.
- Create embeddings/representations that merge clinical notes with contextual data.
The model construction involves two phases:
- Pretraining on an extensive, domain-specific corpus [ref 8].
- Fine-tuning tailored for specific tasks, like classification [ref 9].
After the pretraining phase concludes, the document embedding is introduced to the classifier training. This can be sourced:
- Directly from the output of the pretrained model (document embeddings).
- During the fine-tuning process of the pretrained model. Concurrently, fine-tuning operates alongside active learning for model updates."\
fig. 5 Model weights update with features extraction vs fine tuning
It's strongly advised to utilize one of the pretrained BERT models like ClinicalBERT [ref 10] or GatorTron [ref 11], and then adapt the transformer for classification purposes. However, for this particular project, we initiated BERT's pretraining on a distinct set of clinical notes to gauge the influence of vocabulary and segmentation on prediction accuracy.
Self-attention
Here's a concise overview of the multi-head self-attention model for context:
The foundation of a transformer module is the self-attention block that processes token, position, and type embeddings prior to normalization. Multiple such modules are layered together to construct the encoder. A similar architecture is employed for the decoder.
The foundation of a transformer module is the self-attention block that processes token, position, and type embeddings prior to normalization. Multiple such modules are layered together to construct the encoder. A similar architecture is employed for the decoder.
fig. 6 Schematic for transformer encoder block
Classifier
The classifier is structured as a straightforward feed-forward neural network (fully connected), since a more intricate design might not considerably enhance prediction accuracy. In addition to the standard hyper-parameter optimization, different network configurations were assessed.
The network's structure, including the number and dimensions of hidden layers, doesn't have a significant influence on the overall predictive performance.
The network's structure, including the number and dimensions of hidden layers, doesn't have a significant influence on the overall predictive performance.
Active learning
The goal is to modify models to tackle the issue of covariate shifts observed in the distribution of real-time/production data during inference.
The dual-faceted approach involves:
- Selecting data samples with labels that deviate from the distribution initially employed during training (Active learning) [ref 12].
- Adjusting the transformer for the classification objective using these samples (Transfer learning)
A significant obstacle in predicting diagnostic codes or medical claims is the steep labeling expense. In this context, learning algorithms can proactively seek labels from domain experts. This iterative form of supervised learning is known as active learning.
Because the learning algorithm selectively picks the examples, the quantity of samples needed to grasp a concept is frequently less than that required in traditional supervised learning. In this aspect, active learning parallels optimal experimental design, a standard approach in data analysis [ref 13].
In our scenario, the active learning algorithm picks an unlabeled medical note, termed note-91, and sends it to a human coder who assigns it the diagnostic code S31.623A. Once a substantial number of notes are newly labeled, the model undergoes retraining. Subsequently, the updated model is rolled out and utilized to forecast diagnostic codes on notes in production.
Thank you for reading this article. For more information ...
References
[8] Hugging face: State Of The Art NLP Model Explained
[9] What Is Fine-Tuning and How Does It Work in Neural Networks?
[9] What Is Fine-Tuning and How Does It Work in Neural Networks?
[10] ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
[11] A Large Language Model for Electronic Health Records
[12] Towards data science: Active Learning in Machine Learning
[11] A Large Language Model for Electronic Health Records
[12] Towards data science: Active Learning in Machine Learning
A formal presentation of this project is available at
Glossary
- Electronic health record (EHR): An Electronic version of a patients medical history, that is maintained by the provider over time, and may include all of the key administrative clinical data relevant to that persons care under a particular provider, including demographics, progress notes, problems, medications, vital signs, past medical history, immunizations, laboratory data and radiology reports.
- Medical document: Any medical artifact related to the health of a patient. Clinical note, X-rays, lab analysis results,...
- Clinical note: Medical document written by physicians following a visit. This is a textual description of the visit, focusing on vital signs, diagnostic, recommendation and follow-up.
- ICD (International Classification of Diseases): Diagnostic codes that serve a broad range of uses globally and provides critical knowledge on the extent, causes and consequences of human disease and death worldwide via data that is reported and coded with the ICD. Clinical terms coded with ICD are the main basis for health recording and statistics on disease in primary, secondary and tertiary care, as well as on cause of death certificates
- CPT (Current Procedural Terminology): Codes that offer health care professionals a uniform language for coding medical services and procedures to streamline reporting, increase accuracy and efficiency. CPT codes are also used for administrative management purposes such as claims processing and developing guidelines for medical care review.
---------------------------
Patrick Nicolas has over 25 years of experience in software and data engineering, architecture design and end-to-end deployment and support with extensive knowledge in machine learning.
He has been director of data engineering at Aideo Technologies since 2017 and he is the author of "Scala for Machine Learning" Packt Publishing ISBN 978-1-78712-238-3
He has been director of data engineering at Aideo Technologies since 2017 and he is the author of "Scala for Machine Learning" Packt Publishing ISBN 978-1-78712-238-3
Subscribe to:
Posts (Atom)