Device-to-device (D2D) communication underlaid with cellular networks is a new paradigm, proposed to enhance the performance of cellular networks. By allowing a pair of D2D users to communicate directly and share the same spectral resources with the cellular users, D2D communication can achieve higher spectral efficiency, improve the energy efficiency, and lower the traffic delay. In this paper, we propose a novel joint mode selection and channel resource allocation algorithm via the vertex coloring approach. We decompose the problem into three subproblems and design algorithms for each of them. In the first step, we divide the users into groups using a vertex coloring algorithm. In the second step, we solve the power optimization problem using the interior-point method for each group and conduct mode selection between the cellular mode and D2D mode for D2D users, and we assign channel resources to these groups in the final step. Numerical results show that our algorithm achieves higher sum rate and serves more users with relatively small time consumption compared with other algorithms. Also, the influence of system parameters and the tradeoff between sum rate and the number of served users are studied through simulation results.