Basic question: What is science? The word comes from the Latin "scientia", meaning "knowledge". But science is more than a body of knowledge. Science is something you do and, at heart, doing science means testing ideas. The rest is just bookkeeping. Ask Zombie Feynman.
In reality, of course, science is done with more rigor than the Mythbusters do it. Doing good science requires intelligence, imagination and creativity. Scientists need to make careful observations, come up with good ideas and design crucial experiments. This is an ongoing process that develops increasingly more useful, accurate and comprehensive scientific theories.
"But doesn't scientists need to interpret evidence", you're wondering, "with brains known to be unreliable? What about observer bias?"
Science is not done by lone mad scientists, secluded from the world in their underground lairs. Science is a social enterprise. They cooperate and compete with other people. Also, a scientist's work is scrutinized in the peer-review process and experiments are repeated by other scientists. All to minimize the risk the something went wrong. It's not perfect, science is certainly not the Ultimate Quest for Absolute Truth, but it has this going for it: It works.
"Still", you persist, "what if the world is just one big illusion? Then it doesn't matter how many scientists agree".
Well, that's irrelevant. Science still works within the system; the underlying metaphysics of that system doesn't matter, as long as it's consistent. The fact that matter is mostly empty space doesn't it less real, does it? Would you be less of a person, with thoughts and feelings, if you foung out you were living in the Matrix? If it looks like reality and behaves like reality, it's reality.
Consistency is the one assumption that science make. Consistency is important because today's experiment would be meaningless if tomorrow's physics were different. The consistency can be due to fixed natural laws, particularly lazy gods or just careful programming of the matrix, the nature of reality doesn't actually matter. Luckily, the consistency of reality is a hypothesis that can be tested like any other hypothesis. And every day the natural laws stay the same.