From the perspective of hybridization theory, according to the principle of “linear combination of energy orbitals, most favorable for bonding,” the central atom with d8 should form a four-coordinate structure as shown below.

Thus, dsp² is a hybridization of (n-1)d, ns, and np, and there is no sp²d hybridization because the energy difference is too large. We know that during linear combination, some properties of atomic orbitals will be preserved. I found a supplementary high school chemistry book [1] that states this.

To comment briefly, the “inherited characteristics” of atomic orbitals refer to properties such as symmetry. For example, sp³ hybridization is a combination of s + px + py + pz, making it three-dimensional; sp² hybridization is a combination of s + px + py, making it two-dimensional; and sp hybridization is a combination of s + px, making it one-dimensional.
From the above, we see that dsp² is planar, so the “speculated” orbitals used for hybridization are dx²-y², s, px, py. Therefore, from the perspective of hybridization theory, why can dsp² hybridization not fill the p orbitals? It can be said that it is due to symmetry, as pz cannot participate in hybridization.
However, this explanation seems to have a flavor of circular reasoning. Why must we choose dx²-y²? Why select these four orbitals? Why is it planar? A more “scientific” explanation is similar to the Jahn-Teller distortion explanation; the inorganic chemistry book from the University of Science and Technology of China explains it this way, as shown in the figure below (some books categorize the elongated Jahn-Teller distortion of tetraammine copper ions under dsp² hybridization, which is debatable).
In this theory, due to the disappearance of ligands in the z-direction, orbitals related to z, such as dz², dxz, and dyz, have their energy lowered, which seems to explain why dx²-y² is left empty and also explains why it is a planar configuration, not hybridizing with pz. But it still feels like something is off. The four electron pairs in the valence shell electron pair repulsion model generally form a tetrahedral configuration. Can’t this structure be combined with pz to form a “more stable” tetrahedron? Since there are strong and weak field ligands, can’t the d orbitals be emptied to fill the p orbitals in sp³?

According to Zhang Zude’s “Inorganic Chemistry” P299, dsp² hybridization is quite regular. If we don’t count the Jahn-Teller distortion, which is essentially octahedral for copper ions, it generally involves Ni(II), Pd(II), and Pt(II) with d8, and the total valence electron count is often 16, which is 16e, an exception to the 18-electron rule. Since the valence shell electron pair repulsion model is not very suitable for complexes, we look directly at molecular orbital theory.

In the book “Orbital Interactions in Chemistry, Second Edition (2013)”, the pz (a2u), dxz, dyz (eg), and dxy (b2g) clearly form non-bonding orbitals. The s orbital and dz² combine due to orbital overlap to form 2a1g, likely responsible for “lone pair electrons”. The main bonding orbitals are indeed dx²-y², px, and py. This explains that the considerations of hybridization theory and crystal field theory based on symmetry are correct, without needing the pz orbital. Introducing molecular orbital theory is not just using a cannon to shoot a mosquito; it can explain why d8 16e nickel, palladium, and platinum divalent ion complexes “choose planar square configurations”.

As palladium catalysis and cisplatin research are very extensive, the explanations for these compounds are very detailed. The Walsh diagram above reflects the changes in orbitals during the transformation of the two configurations. The bonding situation of the tetrahedral Td structure on the right can be referenced in crystal field or coordination field theories regarding tetrahedral configurations.
Connecting to the previously mentioned 16-electron complexes, the 2b1g being an empty orbital is quite evident. It is clear that the energy of the left D4h configuration is lower at 16 electrons, which explains why nickel, palladium, and platinum choose planar configurations. It is also worth noting what 2b1g is; it is an antibonding orbital formed by the combination of dx²-y² and four ligands, which is very intuitive. Of course, achieving 18 electrons and choosing tetrahedral structures is also possible, such as in Ni(CO)4. Some older books claim that the configuration on the right is an “sp³ hybridization formed by shifting one orbital back,” which is clearly problematic; anyone who has studied crystal field theory knows why this is incorrect. Are dxz and dyz just there for aesthetics?
Returning to hybridization theory, this hybridization method is indeed the most favorable for bonding. Therefore, the question “Why can dsp² hybridization not fill the p orbitals?” can be answered as “In this structure, when the pz orbital does not participate in hybridization, the system’s energy is lower and more favorable for bonding, thus choosing this structure.”

So can’t we have more electrons to reach 18e or add more ligands? The mechanisms of palladium catalysis and modifications of platinum drugs tell us that it is possible, but then the entire system changes.

Previous answers have also mentioned some issues with s-p-d hybridization theory. Indeed, different explanations can reveal different perspectives. Therefore, when studying chemistry, one must also consider the thought process of the question setter. In short, it can be understood that the electronic configuration of atoms influences the nature of bonding and the selection of stable molecular structures.
References
Ding Zhongyuan. A Brief Explanation of Hybrid Orbital Theory [M]