It's certainly your right to walk off. However, I wanted to point out that what you consider "trivial" is probably not how the person (who is upset) sees it.