Use throughout the following explanation the fact that the area of the
triangle is b*h/2, where b is the length of one side, and h is the
altitude perpendicular to that side. Then if you keep b fixed and
increase h, or vice versa, the area of the triangle will increase.
If any vertex is not on the boundary of the square, you can increase
the area by moving it away from the opposite side, thereby increasing
the altitude. That implies that the triangle with maximum area has all
three vertices on the boundary.
If no vertex is at a corner, then pick any side of the triangle. By
moving the opposite vertex away from this base, you can increase the
area by increasing the altitude. That implies that the triangle with
maximum area has at least one vertex A at a corner.
Suppose that two vertices are on the same side, but not both at a
corner. Then by moving them apart along that side, you can increase the
area by increasing the base between them. Thus if two vertices are on
the same side of the triangle with maximum area, they are at adjacent
corners of the square.
Suppose that neither B nor C is at a corner, and B and C are on
different sides not adjacent to A. Then move B along the side of the
square it is on toward the corner adjacent to the corner A. This will
increase the area by increasing the altitude to side AC, since AC is
not parallel to that side of the square. This implies that the triangle
of maximum area has at least two vertices A and B at adjacent corners.
If the corners A and B are adjacent, then C must be on the side
parallel to AB, and the area is 1/2.
The result is that the largest the area can be is, as you guessed, 1/2,
and that is achieved whenever two vertices are at adjacent corners of
the square and the third vertex is on the opposite side.