I've run into this many times, and although people's reasons might be expressed somewhat differently (if they can express them at all) most of the time it centers around the influence of Evolutionary thinking that has made them uncomfortable with holding to the literal Creation account. In most cases they have become convinced that the earth is millions, even billions of years old, resulting in the need to reject a literal interpretation of Genesis chapters 1-11.
These people usually comfort themselves with the idea that those initial chapters are supposed to be taken allegorically, and that all is well. But because critical thinking is at an all-time low in our culture they don't realize they have shaken the very foundations of all else the Bible teaches.
An example of things that are shaken is the Apostle Paul's statement about the origin of death in Romans 5, which says:
Therefore, just as sin entered the world through one man, and death through sin, and in this way death came to all men, because all sinned... Romans 5:12 (NIV84)
Those who reject a literal interpretation of Genesis and the Creation account are forced to believe that death did NOT enter the world through one man, as Paul wrote, but that death was part of God's original design since they are forced to believe animals were living and dying millions of years before man entered the scene. They must REJECT Paul's statement and all that goes with it.
That's where the extreme danger is: because Paul is making a case in Romans 5 for the God's solution to sin and death, which is Christ dying in our place on the cross.
A person who claims to be a Christians but rejects Genesis (or portions of it) would quickly object to what I'm saying, claiming that they DO believe that Jesus came and died for our sins on the cross. But what they don't realize is that by rejecting Genesis and the Creation account and origin of sin they have eliminated the NEED for Christ's death.
If, as Paul argues in Romans 5, man is NOT responsible for sin, then there was NO NEED for Jesus to come and represent the guilt of man by BECOMING A MAN. The incarnation is now completely unnecessary, because man is NOT responsible for death. GOD IS!
The implications of rejecting a literal Genesis account are absolutely devastating to the Bible's main message, (which is Jesus) and every reference to God one day eliminating death and restoring His creation to its original state (without death) becomes a sham.
And finally, Jesus Himself is made out to be a perpetrator of lies. Since the account of the Global Flood is included in the first 11 chapters of Genesis (and typically considered part of the Bible's allegorical section), and since Jesus spoke of the Flood as an actual historical event, He is made out to be a liar and nothing less (Matthew 24:38).
So, as you can see, it's more than just a slippery slope. It's sinking sand!