Hmmmmmmmm.....I don't think you have a problem. I'm pretty sure I have done much worse and I've been to Canada many many times. Even taking firearms across the border a few times. I doubt they would have extended me 'honors' if there were problems with things 'way back' in my past.
Just do it!