I am not sure what was happening, but every time I would join a new game, hit CTRL + R (gives you ingame FPS) it would be locked at 50FPS, even with the limiter at 61, or even off. To fix it, I had to do the following :
For reference, I have an i5 2500k, 680GTX, 16GB DDR3, 256GB SSD.
Change setting LimitForegroundFPS from 1 to 0
I also reset LimitBackgroundFPS from 1 to 0
This fixed it for me!