Abstract Hyperparameter optimization is very frequently employed in machine learning. However, an optimization of a large space of parameters could result in overfitting of models. In recent studies on solubility prediction the authors collected seven thermodynamic and kinetic solubility datasets from different data sources. They used state-of-the-art graph-based methods and compared models developed for each dataset using different data cleaning protocols and hyperparameter optimization. In our study we showed that hyperparameter optimization did not always result in better models, possibly due to overfitting when using the same statistical measures. Similar results could be calculated using pre-set hyperparameters, reducing the computational effort by around 10,000 times. We also extended the previous analysis by adding a representation learning method based on Natural Language Processing of smiles called Transformer CNN. We show that across all analyzed sets using exactly the same protocol, Transformer CNN provided better results than graph-based methods for 26 out of 28 pairwise comparisons by using only a tiny fraction of time as compared to other methods. Last but not least we stressed the importance of comparing calculation results using exactly the same statistical measures. Scientific Contribution We showed that models with pre-optimized hyperparameters can suffer from overfitting and that using pre-set hyperparameters yields similar performances but four orders faster. Transformer CNN provided significantly higher accuracy compared to other investigated methods.

Igor V. Tetko

Ruud van Deursen

Guillaume Godin

Journal of Cheminformatics

JOURNAL OF CHEMINFORMATICS 是一本开放获取期刊，发表化学信息学和分子建模各个方面的原始同行评议研究。覆盖范围包括但不限于：化学信息系统、软件和数据库，以及分子建模化学结构表示及其在结构中的应用、化学物质和化学反应数据库的子结构和相似性搜索计算机和分子图形、计算机辅助分子设计、专家系统、QSAR 和数据挖掘技术。

The Journal of Cheminformatics is an open-access journal dedicated to original peer-reviewed research in all aspects of cheminformatics and molecular modeling. Its scope includes, but is not limited to: chemical information systems, software and databases, molecular modeling of chemical structure representations and their applications, substructure and similarity searching in chemical substance and reaction databases, computer and molecular graphics, computer-aided molecular design, expert systems, QSAR (Quantitative Structure-Activity Relationship), and data mining techniques.

《Journal of Cheminformatics》是一本开放获取的期刊，专注于化学信息学和分子建模领域的原创性同行评审研究。期刊的覆盖范围包括但不限于：化学信息系统、软件和数据库，以及分子建模中的化学结构表示及其应用、化学物质和化学反应数据库的子结构和相似性搜索、计算机和分子图形、计算机辅助分子设计、专家系统、QSAR（定量构效关系）和数据挖掘技术。

Category	Quartile
化学	2
化学, 化学综合	2
化学, 计算机信息系统	2
化学, 计算机跨学科应用	3

Be aware of overfitting by hyperparameter optimization!

ivySCI AI Smartly Parses PDF, Answers Researchers' Questions, and Helps You Understand Papers in Seconds

Journal Info

Category	Quartile
COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS	1