Yes, on both sides of it. You'd want to do this with the solar cell covered so light doesn't get to it, and next you might want to test again with strong light getting to it, in case the solar cell is the fault, no longer charging it.
The thin blue rimmed battery is probably NiCd, probably...