It's almost impossible to do a precize editing with a pencil/brush. The result is like a kid's drawing or you need a lifetime to do it pixel by pixel. A suitable tool which would determine an object's edges and with settings like AA/color tolerance/whatever would be much better.