When Ben said that everyone had to go back, does that include people like Desmond and Fred?

Here's what I'm thinking, since Ben said "all of you have to go back"-

I'm assuming that not only the Oceanic Six have to go back, but Walt also, since we knew that the island wanted him for a reason (his supposed "powers").

I think that all those from flight 815 that are off island have to go back for some reason...including Walt. I think Desmond did his time protecting the island with all the button pushing year after year.

Desmond is going to have his hands full with Ben trying to kill his one true love