When using the Hand component, all objects under the root of that component will be set to the hand layer, unless they have this component

IgnoreHandLayer.PNG