Manel Martínez-Ramón, PhD, is King Felipe VI Endowed Chair and Professor in the Department of Electrical and Computer Engineering at the University of New Mexico in the United States. He earned his doctorate in Telecommunication Technologies at the Universidad Carlos III de Madrid in 1999.

Meenu Ajith, PhD, is a Postdoctoral Research Associate in Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS) at Georgia State University, Georgia Institute of Technology, and Emory University. She earned her doctorate degree in Electrical Engineering from the University of New Mexico in 2022. Her research interests include machine learning, computer vision, medical imaging, and image processing.

Aswathy Rajendra Kurup, PhD, is a Data Scientist at Intel Corporation. She earned her doctorate degree in Electrical Engineering from the University of Mexico in 2022. Her research interests include image processing, signal processing, deep learning, computer vision, data analysis and data processing.

About the Authors xv

Foreword xvii

Preface xix

Acknowledgment xxi

About the Companion Website xxiii

1 The Multilayer Perceptron 1

1.1 Introduction 1

1.2 The Concept of Neuron 2

1.3 Structure of a Neural Network 14

1.4 Activations 21

1.5 Training a Multilayer Perceptron 22

1.6 Conclusion 37

2 Training Practicalities 41

2.1 Introduction 41

2.2 Generalization and Overfitting 42

2.3 Regularization Techniques 45

2.4 Normalization Techniques 50

2.5 Optimizers 52

2.6 Conclusion 58

3 Deep Learning Tools 61

3.1 Python: An Overview 61

3.2 NumPy 72

3.3 Matplotlib 83

3.4 Scipy 97

3.5 Scikit-Learn 107

3.6 Pandas 116

3.7 Seaborn 125

3.8 Python Libraries for NLP 131

3.9 TensorFlow 138

3.10 Keras 141

3.11 Pytorch 144

3.12 Conclusion 149

4 Convolutional Neural Networks 153

4.1 Introduction 153

4.2 Elements of a Convolutional Neural Network 153

4.3 Training a CNN 160

4.4 Extensions of the CNN 166

4.5 Conclusion 184

5 Recurrent Neural Networks 187

5.1 Introduction 187

5.2 RNN Architecture 188

5.3 Training an RNN 191

5.4 Long-Term Dependencies: Vanishing and Exploding Gradients 199

5.5 Deep RNN 201

5.6 Bidirectional RNN 203

5.7 Long Short-Term Memory Networks 204

5.8 Gated Recurrent Units 218

5.9 Conclusion 221

6 Attention Networks and Transformers 225

6.1 Introduction 225

6.2 Attention Mechanisms 227

6.3 Transformers 242

6.4 BERT 249

6.5 GPT-2 256

6.6.1 Comparison between ViTs and CNNs 264

6.7 Conclusion 269

7 Deep Unsupervised Learning I 273

7.1 Introduction 273

7.2 Restricted Boltzmann Machines 274

7.3 Deep Belief Networks 278

7.4 Autoencoders 279

7.5 Undercomplete Autoencoder 284

7.6 Sparse Autoencoder 285

7.7 Denoising Autoencoders 287

7.8 Convolutional Autoencoder 288

7.9 Variational Autoencoders 291

7.10 Conclusion 297

8 Deep Unsupervised Learning II 301

8.1 Introduction 301

8.2 Elements of GAN 303

8.3 Training a GAN 305

8.4 Wasserstein GAN 309

8.5 DCGAN 312

8.6 cGAN 316

8.7 CycleGAN 318

8.8 StyleGAN 323

8.9 StackGAN 328

8.10 Diffusion Models 333

8.11 Conclusion 338

9 Deep Bayesian Networks 341

9.1 Introduction 341

9.2 Bayesian Models 342

9.3 Bayesian Inference Methods for Deep Learning 344

9.4 Conclusion 352

Problems 353

List of Acronyms 355

Notation 359

Bibliography 365

Index 387