So after some more troubleshooting it looks like it is the Ericsson ONT causing the issues.
With that disconnected from the switch I could ping the server from the workstations while transferring test files back and forth without any issues.
As soon as the ONT is connected back in pings to the server start timing out at random times.
I spoke to the ISP who said that they've come across this issue before.
They want me to write a letter to them about the issues so they can take it to the fibre provider (the company that actually installs the fibre and ont to the customers premises)
Glad I got to the bottom of it!