I've been thinking about this for quite a while, and the concept seems to be popping up more and more recently, so I thought it's time to put the pen to the paper—talk about skeumorphism!—and work through it.
TL;DR: problem setting is a different way to work through problems, and an alternative to problem solving.
Basically, we devote most of our energy towards problem solving. This is because problems either show themselves—e.g. a pipe in our bathroom has burst—or others present us with them—e.g. our app is slow.
We solveacute problems (a pipe bursting) by reacting, and systematically through some sort of a framework or methodology. We've trained ourselves to solve acute problems by learning how to do things, and learning how things work. Acute problems usually come with few unknowns.
In order to solve more complex problems (app slowness) systematically we've had to think about these problems in a different way, because these problems contain a huge amount of unknowns, so we've developed ways to identify these types of problems and ways to solve them, like procedures.
What I have noticed is that when people figure out that there's a problem, they spend little to no time understanding it. They take the problem at face value, and jump into solving mode basically brute-forcing it by throwing any possible solution towards it. Try something, fail, try something else, repeat. A vicious cycle.
Problem setting is about stepping back, and investing energy into describing, and defining a problem so you and others can understand it thoroughly, and then choose a specific method in order to make it go away.
The understanding of the problem prior to solving is what separates this approach. Perhaps even more importantly—in some environments even more so—is enabling this knowledge about the problem to be shared. By sharing what we learned about the problem and sharing it with others, and enabling multiple people to contribute with their own experiences and expertise is what makes this a better approach.
James Heathers believes the term comes from mathematics, but I couldn't find any references to it through shallow research. I did however find a reference to the term in the context of medicine (PDF). Wherever it is mentioned, there's a strong desire to make a distinction between problem solving and setting.
The practice of problem setting is definitely applicable in methodologies such as the scientific method, and perhaps more familiar to us — design thinking, user centered design, etc. A lot of us use it every day without even knowing.