XIRR cares when the money arrived.
A simple gain percentage treats the account like one pot. XIRR treats each deposit and withdrawal as a dated cash flow. Add a large amount near a peak and the result can look much worse than the chart in your head.
That does not mean the calculation is broken. It means it is answering a narrower question: what annualised return did your actual invested cash earn, given the timing?