So I gave up on fixing this issue and was using Desktop capture until recently when I got fed up and went back to testing things.
I managed to fix it and it was a very small change that did the trick.
I ran the RuneScape NXT Client as admin.
Never thought to try it previously.
I closed every program that I have running (I thought f.lux might be the culprit) but Alt1 still won't detect the game if using OpenGL. I checked my NVIDIA settings and to make sure everything was set to Application Controlled and I tried running the game full screen and windowed but nothing fixes it.
Alt1 detects the linked window, and when I turn on "Draw pixel detection debug data on the screen" it puts a red box around the game and a red rectangle around my right clicks in game.
The problem with opengl capture could be a conflict with another capture/overlay program. I know that OBS can cause some problems, it might also be another overlay program. If you're testing different combinations of capture programs keep in mind that changes only take effect after restarting the rs client.
The screenshot you sent does have the correct render settings to work.
I'm having issues getting OpenGL Capture to work, I've tried a number of different things that I've seen suggested in this forum.
My NVIDIA Settings are all set to "Application-controlled"
I've tried with and without "Override Runescape DPI scaling detection"
and I've tried running as admin just to see if that would fix it.
If I make RuneScape run in compatibility mode and use DirectX I can capture just fine but I end up with BSOD due to some unfixed issues with my graphic drivers and windows 10, the only way to solve this is rolling back the drivers which then causes issues in other games.
Basically I need to figure out why OpenGL Capture won't work.
I've attached a screenshot of my ingame ui and graphics settings, if anything else is needed just ask and I'll get it asap.