A maneira clássica seria adicionar vírgulas à esquerda e à direita:
select * from shirts where CONCAT(',', colors, ',') like '%,1,%'
Mas find_in_set também funciona:
select * from shirts where find_in_set('1',colors) <> 0
select * from shirts where CONCAT(',', colors, ',') like '%,1,%'
select * from shirts where find_in_set('1',colors) <> 0
