At what point in history did the Sabbath change from Saturday to Sunday?
Never! The biblical Sabbath was established as part of the Mosaic Covenant between God and Israel and has always been Saturday which is the 7th day of the week.
The obvious follow-up question to this would be: Why then do Christians observe Sunday instead of Saturday as a special day of worship?
Christians gather on Sunday, but they do not “observe” it as a special day.
Early believers gathered on Sunday simply because it was “the Lord’s day”—the day Jesus was raised from the tomb. But nowhere in the Bible are Christians called to observe or “keep” that day—or any other day for that matter.
That doesn't mean the early believers didn't come under fire about this. Obviously some in Colossae were being criticized for not keeping the seventh-day Sabbath, so the Apostle Paul wrote to them saying: "...do not let anyone judge you by what you eat or drink, or with regard to a religious festival, a New Moon celebration or a Sabbath day. These are a shadow of the things that were to come; the reality, however, is found in Christ. Colossians 2:16-17 (NIV84)
The Christians in the region of Galatia were actually bowing to the pressure to observe special days, which prompted the Apostle's following comments: You are observing special days and months and seasons and years! I fear for you, that somehow I have wasted my efforts on you. Galatians 4:10-11 (NIV84)
Why did Paul make these comments about the Sabbath? It's because Sabbath regulations were established between God and Israel. Those regulations do not extend to the Church, which is under a completely different covenant. (See Romans 6:14-15 and Exodus 31:13-17)