"From what I have gathered reading countless enemy discussions, it isn't so much that rightists are implying that "non-white" male refugees should be fighting in the countries they have fled from, but that rightists are accusing them of literally being a de facto army "invading" the countries they are seeking asylum in."
Depends on what degree of racist the rightist is. In the early days of the refugee crisis (2015-2016), I mainly noticed the former argument. Perhaps it has changed.