The idea is to check if the diagonals of the quadrilateral intersect as line segments. See P. Bourke, "Intersection Point of Two Lines (2 Dimensions)" and take note of the last paragraph. The formulas involve determinants; see the Wikipedia entry, "Line-Line Intersection".