The Earl was clear that he wanted all his knights to fully meet their obligations. Failure to do so will have repercussions as the Earl sees fit. Options include fines, a term in the dungeon, or even being stripped of your charter. He may also do nothing - the Earl is not known for being heavy-handed.
You could press your populations for more men, and / or hire mercenaries. That will make them discontent, but they are peasants and cannot be expected to understand the requirements of the kingdom. Also note that knifers are dirt-cheap, and meet the "body count" obligation.
Your troops will be required no matter what you do. If you go with the king, your troops will be placed under the command of another knight, and likely serve garrison, though they might be ordered to the raid without you.