It is important to estimate motion parameters and structures of 3D objects from the image sequence, and a variety of methods to do so have been proposed so far. In this paper, a method is described of estimating the motion and structure of a 3D object from the stereo image sequence. In this method, based on the correspondence of 2D segments in each stereo pair, using a newly devised similarity measure, 3D segments are reconstructed, then grouped into rigid objects, making use of the correspondence of 3D segments in successive images, and parameters of rigid motions are computed for each group.