Believers hold to the doctrine of infallible Scripture because they acknowledge that God is the Creator, Governor, and Sustainer of the universe. Such a God can only speak infallibly.
However, many are hostile to the doctrine of infallibility, and they are usually opposed to God’s governance, too. That’s because they are under the delusion that apart from God, they can, through “science” or state planning, govern the future and speak about their plans with authority. “Hath God said?” leads to “ye shall be as gods,” determining and planning for one’s self what is right and avoiding what is contrary to one’s godless notions of righteousness.
Once you let go of the doctrine of the infallibility of Scripture, Genesis 3:5 becomes your religion.