Nah, it's not cause they are elem or not. Magic energy is based on concentration and use of the mind. Sword or whatever technique are based on concetration and use of the body. Even if the body doesnt attack itself it is the body and not the mind is used to create the attack.

Here a good example Crono Slash is created by concentration and use of the body/sword to attack and Crono Luminaire is an attack that is created by his mind.

If Crono does Fire Whirl his technique is still made by his body. The only reason he does elemental damage is because Lucca imbue Crono sword with fire. Here is another example if someone use a Lightning Sword. The sword technique remain technique. But the damage is magical unless it is a technological electricity.