If someone says ‘that’s hard to swallow’, they are not talking about eating and swallowing food.  The figure of speech means that something is either difficult to believe or difficult to accept.  Maybe a person their lost a job they had for a long time.  A friend might say to that person, ‘that’s hard to swallow’.