I've heard that our phones are not really listening all the time as people have checked, but the search engines and social sites know us really well from how we interact with them. FB may know you better than your partner does. That's a good reason to use it less.
That doesn't mean we shouldn't be careful. There can be malicious apps, but we should be able to control what things they can access. I know Edward Snowden disconnected the microphone on his phone and has to plug in a headset to take calls, but then he knows what the spooks can do.