How do you do this, when the crack is a bit too big for hands & not quite big enough for fists?
i'd say you have funny shaped hands. just kidding, but sorta not really because there's really nothing between hands and fists. What I'm saying is there is overlap and the "manuver" is somewhere between a fist jam and classic hand jam.
it is a little more difficult to explain, but it is the exact same principle as I was talking about before. If you rotate a hand jam far enough towards the thumb it will eventually become a fist jam, only thing laking is you haven't closed your fist yet.
Stick your right hand out in front of you like you were doing a classic hand jam but have your thumb pointing straight up. notice how narrow it is from side to side. now, bring your thumb in front of your palm and try to touch the tip of your thump to your palm where the pinky attaches. see how much wider the profile of you hand is? and see how your thumb muscle bulges out and would be pressing against the left wall of the crack? now rotate your hand 45° twards your thumb. see how the profile of you hand is even wider? Now rotate your hand further until you palm is parallel to ground. This is max width before becoming a classic fist jam. I think I even spread my fingers out sometimes for a little more friction against the walls.
It will help if you stuff this mug up to your elbow, pull it in a little so your elbow is bent and then lean to the left so your fore arm is pressing against left wall of the crack for more friction. It also converts some of the outward pull into resting/leaning against the crack. I dont know if I ever jam without contorting my body one way or the other. Jamming is as much about using your body as it is about your thumb. You're always twisting your arm, leaning, contorting your torso and shoulders or something to make more friction, reach higher or lock down lower.
You might also want to cup your left hand on the edge of the crack like you would if you were offwidthing with your right hand. or pull on an edge or something.
I dont know what the heck you are gonna do for off hands in a pure roof crack situation, except
avoid it and find another option, because the technique doesn't offer nearly as much outward pull as a tight hand jam. I do what I can to reach past this mess or dig deeper into the crack, find a wide place so I stick my leg(s) in ball deep or something. There's always been another option. Who has you following roof cracks that dont fit your hands anyway? I say put them on a #1 camalot sized one and make them suffer as you hike it.
Good luck and have fun.