So it looks like Joe had the upper hand from the 4th move? Ouch...
Yea...
I would not move that knight around so much in the beginning. I think you didn't move anything but the knight for 12 moves, in which Joe could have developed many of his pieces and put them in an attacking position. Meanwhile, the only piece you had developed was a knight.