Isso pode não ser muito bonito, mas acho que vai funcionar:
SELECT cat_id, prod, pos FROM (
SELECT cat_id, pos, prod, if(@last_id = cat_id, @cnt := @cnt + 1, (@cnt := 0 || @last_id := cat_id)) cnt
FROM (
SELECT p.cat_id, pseq.cnt pos, pseq.prod
FROM (
SELECT prod, count(*) cnt FROM prods GROUP BY prod ORDER BY cnt DESC
) pseq
INNER JOIN prods p ON p.prod = pseq.prod
ORDER BY cat_id, pseq.cnt DESC
) po
) plist
WHERE cnt <= 3;
Based on the above data, this will return:
+--------+-----------+-----+
| cat_id | prod | pos |
+--------+-----------+-----+
| 1 | spl php | 2 |
| 1 | kntrn | 1 |
| 1 | kntrn e | 1 |
| 2 | spl php | 2 |
| 2 | zlv | 1 |
| 2 | zlv enter | 1 |
+--------+-----------+-----+
