In general, granular soils are better to build with than clay soils. Sand and gravel will compact better, drain better and often will require less reinforcement. Sands and gravels also have better design properties and this can greatly impact the pressure a retaining wall will deal with. Soils are typically defined by an internal strength characteristic known as a friction angle. This angle is roughly equivalent to the natural angle a pile of this soil would make if it were an oven dry pile of individual particles. Dry clay (with particles not clumped together) will form a much flatter pile naturally than gravel would. This means clay would also apply more pressure to your retaining wall.
If the on-site soils are of a very low quality, you should remove them and replace them with better soils. Using stronger soils will reduce reinforcement requirements, allow faster compaction, and have better long-term performance. Heavy clays and organic soils are both unsuitable for use in the geogrid reinforced zone and should be removed and replaced. If you are unsure about what soils you are working with and you have concerns that they are unsuitable you can always check with a qualified geotechnical engineer to obtain an accurate soil classification.