The 'roamers' aren't siegeing the castle(s).
People that live in castles have servants with them. At least some of them would know how to provide food.
Is that anything like, .”We don’t need farmers, we can just go the grocery store and get food.”?
You can’t grow food, produce electricity, pump water, etc. from inside your castle (home). Or at least 99.9% of the population can’t. That’s why those industries are considered “essential”. Your correct that those people (or any people) aren’t going to “siege” the castle. Fear is. It makes people want to go home and then stay home. It makes government officials order people to go home and to stay there. And for some people, and possibly even some areas of the country, that might be the right thing to do, at least for NOW. But for other people, and other areas, it’s NOT.
In fact, there is real debate about the possibility that for people in urban areas, that live in apartment complexes, it might have been the absolute worst thing to do, as it appears the virus can be transported through ventilation systems and thin walls from one unit to another. In other words, a true castle might actually be the one place you wouldn’t want to stay in. Nursing homes, hospitals, etc. seem to be the hardest hit places, along with some factories, where people stay inside all the time and breath the same recycled air over and over. Even if it’s filtered it can’t remove all of the virus. There are some real interesting articles on this and how other Covid viruses have been spread in the past.