There is a mature literature on what happens when you swap one measuring instrument for another, and it is not in machine learning. The standard treatment is Bland and Altman, "Statistical methods for assessing agreement between two methods of clinical measurement", Lancet 1986; i: 307-310. Their core move is the one worth borrowing: do not correlate the two instruments. Plot their difference against their average , and look at the shape. Correlation is the wrong tool because it measure...