I had 2 significant problems:
I was using DX normal maps when Blender uses OpenGL format. That's bad since they use different coordinate systems.
I was still feeding the Metallic and Roughness in using sRGB as the colour mode. I found out that actually they should be set to nonColor, just like the Normal Map.
So now I have to go back through all the materials I created in this project and fix them. Because I was lazy and didn't set this stuff up properly in the first place. There might be a lesson there somewhere....
https://www.cgbookcase.com/textures/how-to-use-pbr-textures-in-blender