I got an overall score of 110 in the TOEFL, but unfortunately, I just got 23 in the speaking section. Oxford and Cambridge require a minimum score of 25 in each subsection.

Do you think this will affect my application process?

Should I take the test again?

I am afraid that, if I take the test again, I may get a lower overall score.

While I don't have personal experience with this, I know that because these schools are pretty competitive to start with, they tend to be pretty strict with the language requirements. I was just reading about an applicant to LSE, who scored just one point under their TOEFL requirement, and LSE didn't even read the application.

My suggestion would be to contact the application departments at Oxford and Cambridge to see if there's any flex in this requirement.