I work for a major pool builder and see this issue every day.
What you are describing is what is known as Dirt Scale.
Basically, a thin layer of calcium buildup has trapped the dirt on the surface of the pool. This is the reason you cannot brush it away. It's sort of like painting a coat of polyurethane over a spot of dirt on the wall. Unless you break down that calcium buildup, that dirt is not going to go away.
Most people only pay attention to how clear the water looks. If the water is clear, the water chemistry is good. It's not as simple as that, but it is easy.
If you don't already have one, get to your pool supply store and purchase a chemical test kit. The test strips will work, but if you want a more accurate test result, get a kit that includes the drops.
Once you have a test kit, you will want to test the water at least 2-3 times a week. The most important things to pay attention to are PH, Total Alkalinity and Hardness. It is important to keep the PH around 7.2-7.6 to avoid the buildup your seeing.
BTW, there is a very simply way to get rid of that scale. It will take some time but it will work if you put the time and effort into it. All you need to do is drop your PH level to about 6.8-7.0 and BRUSH BRUSH BRUSH. Test the water even more frequently during the process. Like I said, you probably won't notice any improvement right away, but over the course of a few weeks, it will start to fade away.
If you want a "quick fix", you can contact a pool service company (one who knows what they are doing) and they can drain the pool and do an acid wash which will do the same thing as the first method, only faster.