Refactoring is for readability, not for performance.
Performance can and should be measured.
Performance optimization has to come from a data based approach.
Readability cannot be measured. It is therefore even more important to keep improving it, because you can’t prove when it is already good enough.
Refactoring should be part of every software development time plan. From the start.