Usar
HAVING sum(ea.gender = 'female') > 0
AND sum(ea.gender = 'male') > 0
ou
HAVING count(distinct ea.gender) = 2
BTW, você deve usar uma subconsulta para obter todos os dados ao agrupar.
SELECT *
FROM events
where id in
(
SELECT events.id
FROM events
LEFT JOIN event_attendances ON (events.id = event_attendances.event_id)
GROUP BY events.id
HAVING count(distinct event_attendances.gender) = 2
)