Well, you have a point here.

IIRC it depends on the frame of reference. Relative to a magnetic field, the only thing a magnetic force does is changing the velocity direction of affected objects. All work regarding the absolute magnitude of the velocity is zero (no lateral acceleration).