The design and analysis of proximity operations around small solar system bodies is a challenging and complex task. This is mainly due to several destabilizing perturbations such as nonuniform rotating gravity field of the small body, solar radiation pressure, and solar tide. Another important perturbation which is often neglected during the design of close proximity operations is the coupling between orbital and attitude motion of the spacecraft. In most small body missions, the spacecraft is considered to be a point mass as compared to the asteroid, and this assumption has implications on the mission design. This paper compares the dynamics of a rigid body and a point mass in proximity to a small solar system body modeled as a triaxial ellipsoid with uniform density. The small body's gravity field is described using the second degree and order spherical harmonic expansion. The sun's gravity is also considered to act on the asteroid, as well as the point mass and rigid body. A Lie group variational integrator is used to numerically simulate the full (translational and rotational) dynamics of the rigid body. Numerical simulations show the trajectory of the rigid body rapidly diverges from that of the point mass due to the orbit-attitude coupling for small bodies modeled after Near Earth Object 25143 Itokawa. Possible implications on the performance of model-based spacecraft control and on the station-keeping budget if the orbit-attitude coupling is not accounted for in the dynamics model, are presented.