A line used to be scratched in the dirt as a starting line for races, including horse races. Some of the competitors “started from scratch” while others were given a handicap that allowed them to start ahead of the scratch line. And that is how the phrase “starting from scratch” originated.