in general, when the hydraulic fluids flow continuously between the pump and theactuctor(normally motor)without going to the reservoir, then the system is called as closed loop hydraulic system.
On the other hand, if the fluid from the actuator (normally cylinder)first goes to the reservoir and then the fresh fluid from the reservoir flow to the pump inlet,then the system is called the open loop circuit.