Journal of Data and Information Quality

Given a large-scale mobile network with a variety of equipment and radio access network technologies for an approximate 20 million subscribers, there are many types of data that can be used for big data analytics and machine learning (ML) tasks for network operations, monitoring, and optimization. However, a variety of data is measured, collected, and propagated through numerous complex data and software systems. Thus, people, software components, and data-driven operations for big data and ML pipelines face great challenges in dealing with data quality impacts. Data quality related problems occur and are propagated through complex operations involving different types of data, people, software components, and analytics that cannot be solved purely through data quality engineering. This paper discusses our TENSAI framework, as a practical and responsible observability for ensuring data quality in such a mobile network. TENSAI focuses on methods of communication, strategy specifications, and data quality engineering for diverse types of data and analytics among different types of operations. TENSAI presents techniques for capturing and communicating causes/effects of data quality problems clearly to all relevant stakeholders, developing data quality-aware adaptation strategies for actions on data that can be integrated into analytics processes, and engineering data quality awareness in software and data pipelines. Thus, TENSAI supports full visibility of data quality problems and impacts among related systems to empower the utilization and adaptation of data analytics for different types of operations. We will illustrate our TENSAI with several real-world data types, pipelines, and cases based on a real-world mobile network.

Hong-Linh Truong

Ngoc Nhu Trang Nguyen

TENSAI - Practical and Responsible Observability for Data Quality-aware Large-scale Analytics

TENSAI - 数据质量感知的大规模分析的实用且负责任的可观测性

Open Data Observatories refer to online platforms that provide real-time and historical data for a particular application context, e.g., urban/non-urban environments or a specific application domain. They are generally developed to facilitate collaboration within one or more communities through reusable datasets, analysis tools, and interactive visualizations. Open Data Observatories collect and integrate various data from multiple disparate data sources—some providing mechanisms to support real-time data capture and ingest. Data types can include sensor data (soil, weather, traffic, pollution levels) and satellite imagery. Data sources can include Open Data providers, interconnected devices, and services offered through the Internet of Things. The continually increasing volume and variety of such data require timely integration, management, and analysis, yet presented in a way that end-users can easily understand. Data released for open access preserve their value and enable a more in-depth understanding of real-world choices. This survey compares thirteen Open Data Observatories and their data management approaches - investigating their aims, design, and types of data. We conclude with research challenges that influence the implementation of these observatories, outlining some strengths and limitations for each one and recommending areas for improvement. Our goal is to identify best practices learned from the selected observatories to aid the development of new Open Data Observatories.

Naeima Hamed

Omer Rana

Pablo Orozco-terWengel

Benoît Goossens

Charith Perera

A Comparison of Open Data Observatories

开放数据观察站比较研究

Deep learning algorithms have become increasingly popular over the years, having proved their efficiency in input-output functions for distinct types of data. This technology is particularly useful in medical imaging, where complex image structures often generate disagreements between medical staff. These technologies can streamline the diagnostic process by performing automatic image analysis, which results in more accurate and reproducible diagnoses. Additionally, these technologies can enhance content retrieval systems by automatically labeling the images based on the structures they possess. Despite the benefits, the mathematical complexity of deep learning algorithms and their training optimizations can be challenging. Automated machine learning provides a solution to this challenge by offering tools that automate the development and training of these algorithms. This makes it possible for users with limited programming experience to take advantage of these powerful technologies to quickly develop and prototype analysis algorithms for their specific needs. This paper presents a management platform for deep learning services on the cloud that provides a code-free experience through automated machine learning. The evaluation was done in one of the most demanding scenarios, where the service was integrated into a research pathology PACS to annotate mitotic cells in breast cancer tissue automatically. The annotations are processed by an open-source PACS archive and stored directly on the files, enhancing the image metadata and consequently content retrieval systems. The results of the developed algorithms were compared to the state-of-the-art to evaluate the competitiveness of the solution.

Rui Filipe Ribeiro Jesus

Ana Rodrigues

Carlos Costa

Unlocking AutoML: Enhancing Data with Deep Learning Algorithms for Medical Imaging

解锁AutoML：利用深度学习算法增强医学影像数据

Transformer-based models have demonstrated much success in various natural language processing (NLP) tasks. However, they are often vulnerable to adversarial attacks, such as data poisoning, that can intentionally fool the model into generating incorrect results. In this paper, we present a novel, compound variant of a data poisoning attack on a transformer-based model that maximizes the poisoning effect while minimizing the scope of poisoning. We do so by combining the established data poisoning technique (label flipping) with a novel adversarial artifact selection and insertion technique aimed at minimizing detectability and the scope of the poisoning footprint. We find that using a combination of these two techniques, we achieve a state-of-the-art attack success rate (ASR) of \(\sim 90\% \) while poisoning only \(0.5\% \) of the original training set, thus minimizing the scope and detectability of the poisoning action. These findings have the potential to advance the development of better data poisoning detection methods.

Edmon Begoli

Maria Mahbub

Linsey Passarella

Sudarshan Srinivasan

A Compound Data Poisoning Technique with Significant Adversarial Effects on Transformer-based Sentiment Classification Tasks

一种具有显著对抗效果的复合数据中毒技术在基于Transformer的情感分类任务中的应用

The fitness of the systems in which Machine Learning (ML) is used depends greatly on good-quality data. Specifications on what makes a good-quality dataset have traditionally been defined by the needs of the data users—typically analysts and engineers. Our article critically examines the extent to which established data quality frameworks are applicable to contemporary use cases in ML. Using a review of recent literature at the intersection of ML, data management, and human-computer interaction, we find that the classical “fitness-for-use” view of data quality can benefit from a more stage-specific approach that is sensitive to where in the ML lifecycle the data are encountered. This helps practitioners to plan their data quality tasks in a manner that meets the needs of the stakeholders who will encounter the dataset, whether it be data subjects, software developers or organisations. We therefore propose a new treatment of traditional data quality criteria by structuring them according to two dimensions: (1) the stage of the ML lifecycle where the use case occurs vs. (2) the main categories of data quality that can be pursued (intrinsic, contextual, representational and accessibility). To illustrate how this works in practice, we contribute a temporal mapping of the various data quality requirements that are important at different stages of the ML data pipeline. We also share some implications for data practitioners and organisations that wish to enhance their data management routines in preparation for ML.

Maria Priestley

Fionntán O’donnell

Elena Simperl

A Survey of Data Quality Requirements That Matter in ML Development Pipelines

机器学习开发管道中数据质量要求调查

Machine learning (ML) and artificial intelligence (AI) systems rely heavily on human-annotated data for training and evaluation. A major challenge in this context is the occurrence of annotation errors, as their effects can degrade model performance. This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks for three industry-scale ML applications (music streaming, video streaming, and mobile apps). Drawing on real-world data from an extensive search relevance annotation program, we demonstrate that errors can be predicted with moderate model performance (AUC=0.65-0.75) and that model performance generalizes well across applications (i.e., a global, task-agnostic model performs on par with task-specific models). In contrast to past research, which has often focused on predicting annotation labels from task-specific features, our model is trained to predict errors directly from a combination of task features and behavioral features derived from the annotation process, in order to achieve a high degree of generalizability. We demonstrate the usefulness of the model in the context of auditing, where prioritizing tasks with high predicted error probabilities considerably increases the amount of corrected annotation errors (e.g., 40% efficiency gains for the music streaming application). These results highlight that behavioral error detection models can yield considerable improvements in the efficiency and quality of data annotation processes. Our findings reveal critical insights into effective error management in the data annotation process, thereby contributing to the broader field of human-in-the-loop ML.

Heinrich Peters

Alireza Hashemi

James Rae

Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program

可泛化的人类数据标注误差建模：来自大规模行业搜索数据标注项目的证据

短名	J. Data and Information Quality
Journal Impact	1.67
国际分区	COMPUTER SCIENCE, INFORMATION SYSTEMS(Q3)

ISSN	1936-1955, 1936-1963
h-index	29

出版信息	出版商: Association for Computing Machinery (ACM)，出版周期: ，期刊类型: journal，开源期刊: 否
基本数据	创刊年份: 2009，原创研究文献占比: 100.00%，自引率:6.70%， Gold OA占比: 6.90%

Journal of Data and Information Quality

期刊引文格式

有 1 位以上作者的期刊

有 2 位作者的期刊

有 3 位作者的期刊

有 5 位以上作者的期刊

书籍引用格式

学位论文引用格式

网页引用格式

专利引用格式

手工熬夜修改参考文献？研飞自动匹配期刊，一键轻松成稿，支持 Word/WPS

投稿经验分享

TENSAI - Practical and Responsible Observability for Data Quality-aware Large-scale Analytics

A Comparison of Open Data Observatories

Unlocking AutoML: Enhancing Data with Deep Learning Algorithms for Medical Imaging

A Compound Data Poisoning Technique with Significant Adversarial Effects on Transformer-based Sentiment Classification Tasks

A Survey of Data Quality Requirements That Matter in ML Development Pipelines

TENSAI - Practical and Responsible Observability for Data Quality-aware Large-scale Analytics

A Compound Data Poisoning Technique with Significant Adversarial Effects on Transformer-based Sentiment Classification Tasks

Unlocking AutoML: Enhancing Data with Deep Learning Algorithms for Medical Imaging

A Comparison of Open Data Observatories

Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program