A hair is narrow.  It takes more effort than what the effort is worth to try and divide the hair into narrower pieces.  This figure of speech means to disagree about a small issue.  John said ‘100 people came to my party’.  Mary said ‘actually only 97 came’.  John said ‘stop splitting hairs.  The point is that many people came.’