Preface xv
1 Deep Reinforcement Learning Applications in Real-World Scenarios: Challenges and Opportunities 1
Sunilkumar Ketineni and Sheela J.
1.1 Introduction 1
1.1.1 Problems with Real-World Implementation 2
1.2 Application to the Real World 3
1.2.1 Security and Robustness 3
1.2.2 Generalization 5
1.2.2.1 Overcoming Challenges in DRL 9
1.3 Possibilities for Making a Difference in the Real World 11
1.3.1 Transfer Learning and Domain Adaptation 11
1.4 Meta-Learning 12
1.5 Deep Reinforcement Learning (DRL) 13
1.5.1 Hybrid Approaches 14
1.6 Online vs. Offline Reinforcement Learning 15
1.7 Human-in-the-Loop Systems 15
1.8 Benchmarking and Standardization 16
1.9 Collaborative Multi-Agent Systems 18
1.10 Transfer Learning and Domain Adaptation 19
1.11 Hierarchical and Multimodal Learning 21
1.12 Imitation Learning and Human Feedback 22
1.13 Inverse Reinforcement Learning 23
1.14 Sim-to-Real Transfer 24
1.15 Conclusion 25
References 26
2 Deep Reinforcement Learning: A Key to Unlocking the Potential of Robotics and Autonomous Systems 29
Saksham and Chhavi Rana
2.1 Introduction 30
2.1.1 Significance of DRL Field 30
2.1.2 Transformative Advantages of DRL Field 32
2.2 Fields of Investigation 33
2.2.1 General Methods for Investigation 34
2.3 Background 36
2.3.1 Fundamentals of Deep Reinforcement Learning (DRL) 38
2.4 Deep Reinforcement Learning (DRL) in Robot Control 39
2.4.1 Navigation and Localization 40
2.4.2 Object Manipulation 42
2.5 Applications and Case Studies 43
2.6 Challenges and Future Directions 44
2.7 Evaluation and Metrics 46
2.8 Summary 47
References 48
3 Deep Reinforcement Learning Algorithms: A Comprehensive Overview 51
Shweta V. Bondre, Bhakti Thakre, Uma Yadav and Vipin D. Bondre
3.1 Introduction 52
3.1.1 How Reinforcement Learning Works? 53
3.2 Reinforcement Learning Algorithms 53
3.2.1 Value-Based Algorithms 53
3.2.1.1 Q-Learning 53
3.2.1.2 Deep Q-Networks (DQN) 57
3.2.1.3 Double DQN 58
3.2.1.4 Dueling DQN 58
3.3 Policy-Based 59
3.3.1 Policy Gradient Methods 59
3.3.2 REINFORCE (Monte Carlo Policy Gradient) 60
3.3.3 Actor-Critic Methods 61
3.3.4 Natural Policy Gradient Methods 62
3.4 Model-Based Reinforcement Learning 63
3.4.1 Probabilistic Ensembles with Trajectory Sampling (PETS) 63
3.4.2 Probabilistic Inference for Learning Control (PILCO) 64
3.4.3 Model Predictive Control (MPC) 65
3.4.4 Model-Agnostic Meta-Learning (MAML) 66
3.4.5 Soft Actor-Critic with Model Ensemble 67
3.4.6 Deep Deterministic Policy Gradients with Model (DDPG with Model) 68
3.5 Characteristics of Reinforcement Learning 69
3.6 DRL Algorithms and Their Advantages and Drawbacks 71
3.7 Conclusion 72
References 72
4 Deep Reinforcement Learning in Healthcare and Biomedical Applications 75
Balakrishnan D., Aarthy C., Nandhagopal Subramani, Venkatesan R. and Logesh T. R.
4.1 Introduction 76
4.2 Related Works 76
4.3 Deep Reinforcement Learning Framework 80
4.4 Deep Reinforcement Learning Applications in Healthcare and Biomedicine 81
4.5 Deep Reinforcement Learning Employs Efficient Algorithms 82
4.5.1 Deep Q-Networks 82
4.5.2 Policy Differentiation Techniques 82
4.5.3 Hindsight Experience Replay (HER) 82
4.5.4 Curiosity-Driven Exploration 82
4.5.5 Long Short-Term Memory Networks and Recurring Neural Network Designs 82
4.5.6 Multi-Agent DRL 83
4.6 Semi-Autonomous Control Based on Deep Reinforcement Learning for Robotic Surgery 83
4.6.1 Double Deep Q-Network (DDQN) 83
4.6.2 Materials and Methods 84
4.6.3 Results 86
4.6.4 Discussion 87
4.7 Conclusion 87
References 88
5 Application of Deep Reinforcement Learning in Adversarial Malware Detection 91
Manju and Chhavi Rana
5.1 Introduction 91
5.1.1 Background 95
5.1.2 Significance of Malware Detection 96
5.1.3 Challenges with Adversarial Attacks 96
5.2 Foundations of Deep Reinforcement Learning 97
5.2.1 Overview of Deep Reinforcement Learning 98
5.2.2 Core Concepts and Components 99
5.2.3 Relevance to Malware Detection 100
5.3 Malware Detection Landscape 101
5.3.1 Evolution of Malware Detection Techniques 102
5.3.2 Adversarial Attacks in Cybersecurity 103
5.3.3 Need for Advanced Detection Strategies 104
5.4 Deep Reinforcement Learning Techniques 104
5.4.1 Application of Deep Learning in Malware Detection 105
5.4.2 Reinforcement Learning Algorithms 106
5.5 Feature Selection Strategies 107
5.5.1 Importance of Feature Selection in Malware Detection 108
5.5.2 Techniques for Feature Selection 108
5.5.3 Optimization for Deep Reinforcement Learning Models 109
5.6 Datasets and Evaluation 110
5.7 Generating Adversarial Samples 111
Conclusion and Future Directions 112
Future Directions 112
References 112
6 Artificial Intelligence in Blockchain and Smart Contracts for Disruptive Innovation 115
Eashwar Sivakumar, Kiran Jot Singh and Paras Chawla
6.1 Introduction 115
6.1.1 Smart Contract 116
6.2 Literature Review 117
6.2.1 Blockchain and Smart Contracts in Digital Identity 117
6.2.2 Blockchain and Smart Contracts in Financial Security 118
6.2.3 Blockchain and Smart Contracts in Supply Chain Management 119
6.2.4 Blockchain and Smart Contracts in Insurance 120
6.2.5 Blockchain and Smart Contracts in Healthcare 121
6.2.6 Blockchain and Smart Contracts in Agriculture 121
6.2.7 Blockchain and Smart Contracts in Real Estate 122
6.2.8 Blockchain and Smart Contracts in Education and Research 123
6.2.9 Blockchain and Smart Contracts in Other Sectors 124
6.3 Critical Analysis of the Review 125
6.4 Blockchain and Artificial Intelligence 128
6.5 Discussion on the Reasoning for Implementation of Blockchain 129
6.6 Conclusion 130
References 130
7 Clinical Intelligence: Deep Reinforcement Learning for Healthcare and Biomedical Advancements 137
Keerthika K., Kannan M. and T. Saravanan
7.1 Introduction 138
7.2 Deep Reinforcement Learning Methods 138
7.2.1 Model-Free Methods 138
7.2.2 Policy Gradient Methods 139
7.2.3 Model-Based Methods 139
7.3 Applications of DRL in Healthcare 140
7.3.1 Tailored Treatment Recommendations 140
7.3.2 Optimization of Clinical Trials 141
7.3.3 Disease Diagnosis Support 142
7.3.4 Accelerated Drug Discovery and Design 142
7.3.5 Enhanced Robotic Surgery and Assistance 142
7.3.6 Health Management System 143
7.4 Challenges 143
7.5 Healthcare Data Types 144
7.5.1 Electronic Healthcare Records (EHRs) 144
7.5.2 Laboratory Data 145
7.5.3 Sensor Data 145
7.5.4 Biomedical Imaging Information 145
7.6 Guidelines for the Application of DRL 147
7.7 A Case Study: DRL in Healthcare and Biomedical Applications 147
7.7.1 Optimizing Radiation Therapy Dose Distribution in Cancer Treatment 147
7.7.2 Dose Strategy Model in Sepsis Patient Treatment 148
References 149
8 Cultivating Expertise in Deep and Reinforcement Learning Principles 151
Chilakalapudi Malathi and J. Sheela
8.1 Introduction 151
8.1.1 Reinforcement Learning's Constituent Parts 152
8.1.2 Process of Markov Decisions (MDP) 152
8.1.3 Learning Reinforcement Methods 153
8.2 Intensive Learning Foundations 164
8.2.1 A Definition of Deep Learning 164
8.2.2 Deep Learning Elements 164
8.2.2.1 Different Kinds of Deep Learning Networks 165
8.3 Integrating Deep Learning and Reinforcement Learning 172
8.3.1 Deep Reinforcement Learning 172
8.3.2 Deep Reinforcement Learning Complexity Problems 174
Conclusion 175
References 175
9 Deep Reinforcement Learning in Healthcare and Biomedical Research 179
Shruti Agrawal and Pralay Mitra
9.1 Introduction 180
9.1.1 Reinforcement Learning 180
9.1.2 Deep Reinforcement Learning 181
9.2 Learning Methods in Bioinformatics with Applications in Healthcare and Biomedical Research 182
9.2.1 Protein Folding 182
9.2.2 Protein Docking 183
9.2.3 Protein-Ligand Binding 185
9.2.4 Binding Peptide Generation 187
9.2.5 Protein Design and Engineering 188
9.2.6 Drug Discovery and Development 190
9.3 Applications in Biological Data 192
9.3.1 Omics Data 192
9.3.2 Medical Imaging 192
9.3.3 Brain/Body-Machine Interfaces 193
9.4 Adaptive Treatment Approach in Healthcare 193
9.5 Diagnostic Tools in Healthcare and Biomedical Research 195
9.6 Scope of Deep Reinforcement Learning in Healthcare and Biomedical Applications 196
9.6.1 State and Action Space 196
9.6.2 Reward 197
9.6.3 Policy 198
9.6.4 Model Training 199
9.6.5 Exploration 199
9.6.6 Credit Assignment 200
9.7 Conclusions 200
References 201
10 Deep Reinforcement Learning in Robotics and Autonomous Systems 207
Uma Yadav, Shweta V. Bondre and Bhakti Thakre
10.1 Introduction 208
10.2 The Promise of Deep Reinforcement Learning (DRL) in Real-World Robotics 210
10.3 Preliminaries 211
10.4 Enhancing RL for Real-World Robotics 222
10.5 Reinforcement Learning for Various Robotic Applications 224
10.6 Problems Faced in RL for Robotics 231
10.7 RL in Robotics: Trends and Challenges 232
10.8 Conclusion 235
References 236
11 Diabetic Retinopathy Detection and Classification Using Deep Reinforcement Learning 239
H.R. Manjunatha and P. Sathish
11.1 Introduction 239
11.2 Literature Survey 243
11.3 Diabetic Retinopathy Detection and Classification 248
11.4 Result Analysis 256
11.5 Conclusion 260
References 260
12 Early Brain Stroke Detection Based on Optimized Cuckoo Search Using LSTM¿Gated Multi-Perceptron Neural Network 265
Anita Venaik, Asha A., Dhiyanesh B., Kiruthiga G., Shakkeera L. and Vinodkumar Jacob
12.1 Introduction 266
12.2 Literature Survey 268
12.2.1 Problem Statement 269
12.3 Proposed Methodology 270
12.3.1 Dataset Collection 270
12.3.2 Preprocessing 271
12.3.3 Genetic Feature Sequence Algorithm (GFSA) 275
12.3.4 Disease-Prone Factor (DPF) 281
12.3.5 Decision Tree-Optimized Cuckoo Search (DTOCS) 284
12.3.6 Long Short-Term Memory Gate Multilayer Perceptron Neural Network (LSTM-MLPNN) 289
12.4 Result and Discussion 293
12.4.1 Performance Matrix 293
12.5 Conclusion 296
References 297
13 Hybrid Approaches: Combining Deep Reinforcement Learning with Other Techniques 301
M. T. Vasumathi, Manju Sadasivan and Aurangjeb Khan
13.1 Introduction 302
13.1.1 Digital Twin-Introduction 302
13.1.2 Model of a Digital Twin 302
13.1.2.1 Steps Involved in Building a Digital Twin Prototype 303
13.1.3 Application Areas of Digital Twins 303
13.1.3.1 Digital Twin in Medical Field 304
13.1.3.2 Digital Twin in Smart City 304
13.1.3.3 Digital Twin in Sports 304
13.1.3.4 Digital Twin in Smart Manufacturing 305
13.2 Digital Twin Technologies 305
13.2.1 Data Acquisition and Sensors 306
13.2.2 Data Analytics and Machine Learning 306
13.2.3 Cloud Computing 307
13.2.4 Other Technologies 307
13.3 Integration of RL and Digital Twin 307
13.3.1 Motivation for Combining Digital Twin and RL 309
13.3.2 How RL Enhances Decision-Making Within Digital Twins 310
13.4 Challenges of Using RL in Digital Twins 311
13.5 Digital Twin Modeling with RL 312
13.6 Technology Underlying RL-Based Digital Twins 314
13.6.1 Integration of RL with Digital Twins in Four Stages 314
13.6.2 Tools and Libraries for Developing RL-Based Digital Twins 314
13.6.2.1 Simulation and Digital Twin Platforms 314
13.6.2.2 Reinforcement Learning Libraries 315
13.6.3 Integration with Existing Systems and IoT Devices for RL Deployment 315
13.6.3.1 Data Collection and Sensor Integration 315
13.6.3.2 Communication and Data Ingestion 316
13.6.3.3 Digital Twin Integration 316
13.6.3.4 RL Integration 316
13.6.3.5 Control and Actuation 316
13.6.3.6 Implementation of Feedback and Learning Process 316
13.6.3.7 Dashboard for Alert and Visualization 316
13.6.3.8 Ensuring the Security and Authentication 317
13.7 Industry-Specific Applications: A Case Study of DT in a Car Manufacturing Unit 317
13.7.1 IoT Components Required for Creating Digital Twin for the Manufacturing Unit 318
13.7.2 Architecture of the Proposed Digital Twin for Car Manufacturing Unit 318
13.7.3 Challenges and Opportunities in the Implementation of DTs for Car Manufacturing 320
13.8 Conclusion 321
References 322
14 Predictive Modeling of Rheumatoid Arthritis Symptoms: A High-Performance Approach Using HSFO-SVM and UNET-CNN 325
Anusuya V., Baseera A., Dhiyanesh B., Parveen Begam Abdul Kareem and Shanmugaraja P.
14.1 Introduction 326
14.1.1 Novelty of the Research 327
14.2 Related Work 328
14.2.1 Challenges and Problem Identification Factor 331
14.3 HSFO-SVM Based on LSTM-Gated Convolution Neural Network (lstmg-cnn) 332
14.3.1 C-Score and Cross-Fold Validation 332
14.3.2 Honey Scout Forager Optimization 335
14.3.3 Feature Selection Using SVM 336
14.3.4 UNET-CNN Classification 338
14.4 Result and Discussion 341
14.5 Conclusion 345
References 346
15 Using Reinforcement Learning in Unity Environments for Training AI Agent 349
Geetika Munjal and Monika Lamba
15.1 Introduction 349
15.2 Literature Review 351
15.3 Machine Learning 352
15.3.1 Categorization of Machine Learning 352
15.3.1.1 Supervised Learning 352
15.3.1.2 Unsupervised Learning 353
15.3.1.3 Reinforcement Learning 353
15.3.2 Classifying on the Basis of Envisioned Output 353
15.3.2.1 Classification 354
15.3.2.2 Regression 354
15.3.2.3 Clustering 354
15.3.3 Artificial Intelligence 354
15.4 Unity 354
15.4.1 Unity Hub 355
15.4.2 Unity Editor 355
15.4.3 Inspector 355
15.4.4 Game View 355
15.4.5 Scene View 355
15.4.6 Hierarchy 355
15.4.7 Project Window 356
15.5 Reinforcement Learning and Supervised Learning 356
15.5.1 Positive Reinforcement 357
15.5.2 Negative Reinforcement 357
15.5.3 Model-Free and Model-Based RL 357
15.6 Proposed Model 359
15.6.1 Setting Up a Virtual Environment 359
15.6.2 Setting Up of the Environment 360
15.6.2.1 Creating and Allocating Scripts for the Environment 361
15.6.2.2 Creating a Goal for the Agent 361
15.6.2.3 Reward-Driven Behavior 361
15.7 Markov Decision Process 362
15.8 Model-Based RL 362
15.9 Experimental Results 363
15.9.1 Machine Learning Models Used for the Environments 363
15.9.2 PushBlock 363
15.9.3 Hallway 365
15.9.4 Screenshots of the PushBlock Environment 368
15.9.5 Screenshots of the Hallway Environment 369
15.10 Conclusion 372
References 372
16 Emerging Technologies in Healthcare Systems 375
Ravi Kumar Sachdeva, Priyanka Bathla, Samriti Vij, Dishika, Madhur Jain, Lokesh Kumar, G. S. Pradeep Ghantasala and Rakesh Ahuja
16.1 Introduction 375
16.2 Personalized Medicine 376
16.3 AI and ML in Healthcare Sector 377
16.3.1 AI in Medical Diagnosis 378
16.3.2 Drug Discovery 378
16.3.3 Personalized Treatment Plans 379
16.3.4 Pattern Matching or Trend Detection 380
16.4 Immunotherapy 380
16.4.1 Monoclonal Antibodies 381
16.4.2 Checkpoint Inhibitors 381
16.4.3 CAR-T Cell Therapy 381
16.5 Regenerative Medicine 381
16.6 Digital Health (Use of Technology in Healthcare) 383
16.6.1 Wearable Devices 383
16.6.2 Telemedicine 384
16.6.3 Electronic Health Records 384
16.7 Health Inequity 385
16.7.1 Health Disparity 385
16.7.2 Health Equity 385
16.8 Future Directions in Healthcare Research 385
16.9 Challenges and Recommendations for Advanced Level of Modern Healthcare Technologies 386
16.9.1 Challenges 387
16.9.2 Recommendations 388
16.10 Healthcare Sector in Developing and Underdeveloped Countries 388
16.10.1 Healthcare Sector in Developing Countries 388
16.10.2 Healthcare Sector in Underdeveloped Countries 389
16.11 Comparison of Recent Progress and Future Mentoring in Healthcare Using Technology 389
16.12 Conclusion 391
References 392
Index 395
