Use
insert . . . select
:INSERT INTO flow_instances (custom_id_year, custom_id_counter)
WITH this_year AS (
SELECT YEAR(CURDATE()) as this_year
),
max_val AS (
SELECT COALESCE(MAX(custom_id_counter), 0) as max_val
FROM flow_instances AS max_val
WHERE custom_id_year = YEAR(CURDATE())
)
SELECT ty.this_year, mv.max_val + 1
FROM this_year ty CROSS JOIN
max_val mv;
Você precisa fazer referência aos CTEs para usar os valores que eles definem.