Almost all manually collectable plants larger than growable crops, have broken or irregular hitboxes, this results in you not being able to tear down what's directly in front of you, and instead having to play pick the potato to see which plants hitbox is overlapping with the other and then trying to tear that one down instead of the one you wanted to originally tear down. Larger plants like the tearable trees don't seem to have this issue unless the mutant shrubbery is around.

A potential fix I can think of is make a new hitbox at the base of the plants in question to effectively tear them down by the base of the roots, but I'm not a coder or programmer so I really shouldn't talk.