Condicional na cláusula where do MYSQL

Você pode fazer isso com AND e OR contanto que você use parênteses suficientes.

Também estou assumindo que activities.OrderBy pode ser nulo. Se não for esse o caso, você pode remover a verificação nula:

SELECT activities.*, 
       activitytypes.orderby 
FROM   activities 
       LEFT OUTER JOIN activitytypes 
                    ON activities.typeid = activitytypes.typeid 
WHERE  activities.userid = 86 
       AND activities.typeid NOT IN ( 5, 10, 11, 12, 19 ) 
       AND ( ( activities.orderby = 1 
               AND activities.starttime >= '2013-08-26 04:00:00' 
               AND activities.endtime <= '2013-08-27 04:00:00' ) 
              OR ( ( activities.orderby IS NULL 
                      OR activities.orderby != 1 ) 
                   AND activities.activitydate = '2013-08-26' ) ) 
ORDER  BY activitytypes.orderby, 
          activities.starttime

Alternativamente, se você ainda quiser usar CASE , você só precisa fechar seu CASE declaração usando END , assim:

SELECT          activities.*, 
                activitytypes.orderby 
FROM            activities 
LEFT OUTER JOIN activitytypes 
ON              activities.typeid = activitytypes.typeid 
WHERE           activities.userid = 86 
AND             activities.typeid NOT IN ( 5, 10, 11, 12, 19 ) 
AND             ( 
                  CASE 
                  WHEN activities.orderby = 1 THEN
                    activities.starttime >= '2013-08-26 04:00:00' AND activities.endtime <= '2013-08-27 04:00:00'
                  ELSE
                    activities.activitydate = '2013-08-26' 
                  END
                ) 
ORDER BY        activitytypes.orderby, 
                activities.starttime