As far as the external appearance is concerned both behaviour, patience and cowardice, seem similar. But internally they are quite different. Patience is a principled behaviour while cowardice is a behaviour which is based on sheer interest. If you are satisfied that what you are doing is based on a well-considered principle, then, your case is a case of patience rather than a case of cowardice. Don't bother if people call you a coward.