Abstract: The ground-based transient electromagnetic method (TEM), with unique advantages, has been extensively applied for hydrogeological exploration in coal mines. However, in field exploration, complex terrains tend to cause variations in geometric shapes of transmitting sources and deviations in signals received, reducing exploration accuracy. Hence, by integrating theoretical analysis and numerical simulations with the analysis of actual terrain data, this study systematically investigated the response characteristics of electrical- and magnetic-source TEM fields under complex terrains. Using the vector finite-element electromagnetic forward modeling technique based on second-order backward Euler discretization, this study delved into the laws of influence of peak and valley terrains as locations of receiving points and transmitting sources on the two source systems. The results show that the electrical-source system suffered signal changes, with responses being enhanced in the case of the source wire position higher than the surrounding terrain and weakened otherwise. In contrast, the magnetic-source system manifested no signal changes but similar response variations. Then, based on measured terrain data, this study established a three-dimensional forward model that highly matched actual terrains using tetrahedral grids. The simulation results indicate that the earlier response curves can distinctly mirror the terrain details, with curve morphologies consistent with terrain features, whereas the later response curves gradually tended to flatten, suggesting the influence of deep structures. The findings of this study will provide theoretical guidance for exploration device arrangement, as well as data acquisition, processing, and interpretations, of the TEM method under complex terrains, thereby holding critical significance for improving the hydrogeological exploration accuracy of coal mines.