My boyfriend's fishing hobby (he's 23) is starting to annoy me (I'm 23). Am I being unreasonable?

My boyfriend and I have been together for three years now. Our lives are quite busy, and we usually spend time together from Sunday to Friday. We share meals and sleep together before I head off to my internship and he goes to work. On Friday nights, I go to my parents’ house, which is about an hour away, to work on Saturday. I often return on Saturday night, but sometimes I come back on Sunday. There have been instances where his fishing habits have caused some tension in our relationship. When I bring it up, he often responds with, "Yeah, but..." and I usually end up letting it go. Fortunately, he has been fishing less frequently in recent years, so it hasn't been as much of an issue for me. However, lately, he has started making more plans to go fishing again. Since he works on Saturdays, he tries to find some free time during the week, but unfortunately, it never aligns with my schedule. For several weeks, he has promised to look for a day when we can hang out together, but I’ve been left waiting, hoping we can enjoy some time as a couple. Today, I'm home sick, and yesterday he told me he was free today and had already arranged to go fishing. I said that was fine, but now I’ve found out he won't be back until after dinner. He left early this morning at 6, and I'm feeling irritated. I haven’t heard from him all day and feel uncertain about where we stand. I had hoped he would return a bit earlier, even though I recognize that being sick limits what we could do together. Still, I'm disappointed, especially after waiting so long for that promised time together. Am I being unreasonable for feeling this way? I'm starting to doubt myself, especially since I haven't been feeling like myself lately.