A Test for the Convexity of a Quadrilateral
This Demonstration draws the quadrilateral green if convex, orange if concave, and red if reflex.
The idea is to check if the diagonals of the quadrilateral intersect as line segments. See P. Bourke, "Intersection Point of Two Lines (2 Dimensions)" and take note of the last paragraph. The formulas involve determinants; see the Wikipedia entry, "Line-Line Intersection".