Are they close enough that it would be unforgivable if you miss their 50th? Most people would consider missing their 50th, for a convention, unacceptable. It is one of those big life events. As much fun as fanime can be, is it worth possibly losing an relationship?
If they are distant enough, they may not care if you don't show.
Is it possible enough to do both? If you leave San Jose on May 22 or May 23 (depending on the time of the event), you could probably do both. Monday is pretty much a dead day anyway. Just give yourself enough time, rest & maybe a car buddy? You can probably talk to hotel staff about leaving early.
If you need to, skip fanime. Though that'd be a last resort for me. I'll be driving 6 hours to attend. XD