By Hitesh Raval, Principal Data Scientist & CDISC SME
When submitting clinical trial data to regulatory agencies such as the FDA (U.S.) and PMDA (Japan), it is crucial to follow submission data standards, such as CDISC standards to ensure compliance. The manual mapping of data fields from a clinical trial database to the corresponding compliant standards, like CDISC-SDTM, has resulted in a significant increase in statistical programming resources required for a clinical trial. To address this issue, GenInvo has leveraged use of Machine Learning to develop tool (ApoGI-DT™) that helps in automating this process, reducing the time and resources needed to develop high-quality, standards-compliant datasets (such as SDTM) for regulatory submission.
Traditional approach to standard compliant data transformation
The traditional approach of standard compliant data transformation (such as CDISC-SDTM) typically involves manual and rule-based procedures that are not only time-consuming but also prone to errors, necessitating specialized technical expertise. This approach frequently entails the development of customized programs to extract data from various sources, manually mapping the raw clinical data to standards-compliant data (such as CDISC-SDTM) domains and elements and validating the data against the established standards. Below mentioned are few of the current challenges in mapping process:
- Laborious and time-consuming
- Manual processing of codelist mapping, unit conversion and other similar task
- Manual errors concerning the accuracy and completeness.
- Inconsistency in mapping across studies
- Necessity to understand raw data/metadata as precursor to manual mapping.
Automating the data transformation with Natural Language Processing:
Natural Language Processing along with Human in Loop approach (HIL) offers an opportunity to automate and optimize data transformation to generate standards-compliant datasets (such as CDISC-SDTM) for regulatory submission. This can be achieved through the below mention steps.
Standards-Compliant Dataset Generation Flow:
Mapping Automation
Utilizing modern Machine Learning and Pattern Recognition Algorithms, variable mapping has become as seamless as it can ever be.
- ML Model Training: The model is trained using existing standards-compliant datasets (such as CDISC-SDTM) and their associated metadata. Throughout the training process, the model acquires knowledge of the relationships and patterns between the raw clinical trial data and the corresponding standard domains, variables, and values. This training enables the model to comprehend how various variables in the raw data align with specific standards-compliant data (such as CDISC-SDTM) components.
- Provide Input Raw Data: The ML model is exceptionally generalized and can accept raw data in various formats such as spreadsheets, databases or any other structured data format.
- Automated Mapping to output Standard Data: The model leverages its acquired knowledge in autonomously mapping the raw data variables with the relevant standards-compliant data (such as CDISC-SDTM) domains and variables. Through the analysis of learned patterns and associations during the training process, the model accurately identifies the corresponding standards-compliant data (such as CDISC-SDTM) components for each variable within the raw data and finally outputs the mapped standards-compliant datasets (such as CDISC-SDTM) based on the input raw data.
Through the automation of the mapping process, the model effectively minimizes the need for manual intervention and mitigates the risk of human errors. This automated mapping capability proves especially advantageous for studies encompassing extensive datasets or projects that frequently undergo updates to the source structure.
Nevertheless, it is essential to acknowledge that while AI offers a substantial advantage in mapping standards-compliant data (such as CDISC-SDTM), human review/validation remains indispensable. With Human in Loop approach (HIL), Domain Experts review the generated mappings to ensure precision and make any essential modifications based on their domain expertise and the specific demands of their therapeutic areas.
Validating and Standardizing Data
In the process of generating standards-compliant data (such as CDISC-SDTM), data validation and standardization are crucial steps that guarantee accuracy, consistency, and reliability of the data used in the study. AI models can play a significant role in automating and improving these steps when working with clinical data standards (such as CDISC).
- Error Identification: By harnessing the power of machine learning algorithms, AI models have the capability to swiftly analyse standards-compliant datasets (such as CDISC-SDTM) and detect potential errors or inconsistencies present within the data. These errors may include missing values, incorrect date formats, and other similar issues.
- Inconsistency Resolution: Through the analysis of patterns and relationships within standard data, AI models has the capability to detect instances where data entries exhibit contradictions. An illustrative example would be if a patient’s death information is inconsistent across standard domains (like AE, DM, DD domains of SDTM). In such cases, AI models can promptly flag these potential inconsistencies, drawing attention to the need for further investigation. By automating the identification of such inconsistencies, AI plays a crucial role in upholding the integrity and reliability of the data.
- Missing Data Handling: AI models have the capability to utilize data query techniques to identify missing values by analysing the existing data patterns. This functionality proves valuable for statistical programmers and data managers as it enables them to proactively raise queries and expedite the data cleaning process. By leveraging this feature, data cleaning can be accomplished more efficiently and effectively.
- Data Standardization: Clinical data standards (such as CDISC SDTM) provide standardized structures and variable definitions for clinical trial data. However, when working with real-world data, inconsistencies in data representation can arise. AI models offers valuable assistance in standardizing the data by aligning it with the clinical data standards. It possesses the capability to map and convert the data, ensuring adherence to the standards and promoting consistency and compatibility.
AI models plays a crucial role by automating the detection of errors, resolving inconsistencies, handle missing data and ensuring compliance with clinical data standards. By leveraging the capabilities of machine learning and pattern recognition, AI provides valuable assistance and enabling to generate standards-compliant datasets (such as CDISC-SDTM) that is of high quality and reliable.
Conclusion
The utilization of Clinical Data Standards & Machine Learning techniques along with Human in loop (HIL) approach for clinical data transformation is emerging as promising solution to address the challenges of traditional manual processes. It offers significant advantages in terms of efficiency, accuracy and standardization. ML empowers researchers and data managers to optimize standards-compliant data (such as CDISC-SDTM) generation processes. However, human oversight and interaction remain critical to ensure the accuracy and compliance of the generated standards-compliant datasets (such as CDISC-SDTM). Additionally, these approaches can handle various data formats and sources, making them more flexible and adaptable to the variability in clinical trial data. With automation through AI, Statistical programmers can manage more studies and free up team members for higher-value activities which ultimately accelerate the entire process of clinical data reporting.
We, At GenInvo developed tool that can help you to accelerate generating standards-compliant datasets (such as CDISC-SDTM) by reducing manual efforts and automating the entire process. Looking for Better Option to transform your clinical data to standards-compliant data, visit our product ApoGI-DT™
