logoalt Hacker News

plaidfujitoday at 3:32 AM0 repliesview on HN

In chemicals and materials, 50 rows of good data is a really solid study. That’s e.g. a 3x4x4 experimental design (assuming replicates for each condition get averaged into a single row). If you managed to prep that many samples correctly and obtain consistent characterization data across all properties of interest, you’ve easily got a paper. It’s also kind of malpractice to jam this type of data (few samples, wide rows) into modern ML models. There are plenty of simpler statistical methods that will tell you what’s going on, and even then a well-made plot might be good enough. The difficulty is not in drawing insight from the final numbers, it’s almost always in how those numbers came to be in the first place.

Thus the reticence of science-oriented companies to invest heavily in these mass data-gathering exercises to feed ML. It’s damn expensive, and almost always leads you back to raw data issues, not breakthrough discovery. Doing it without a set purpose in mind is even more likely to yield garbage.