Not at all - I think you are observing that the people who occasionally get emails delayed are using gmail. But without an understanding of how many users are on gmail that doesn't mean that gmail is the issue.
It's just that the people who don't see delays don't tell you what email system they are using... so you never know that they are running gmail as well.
In general gmail is both reliable and prompt.
Who is your supplier? Unless they are one of the giants then I'd be more likely to suspect that they had the occasional smtp hiccup.