So, in my view, the Japanese received swift, terrible but well advised retribution. This was interpreted by the Allies in the first sense, which they had the right to do, since the whole point of Potsdam was that the Japanese administration no longer had any rights whatsoever. The official response of the Japanese was that of " mokusatsu", which can be interpreted in a very negative way, "to kill with silence", or can mean only "to practise wise inactivity". On July 26, the United States, Britain, and China released the Potsdam Declaration announcing the terms for Japan's surrender, with the warning, " We will not deviate from them. Japan, because of its continuing intransigence, got itself even more crushed and defeated because of actions and decisions it had made to the Potsdam declarations.
I might add also that the the Russians also unleashed all the wepons at their disposal, a somewhat less technologically impressive weapon, but a wepon of great power nevertheless.
Truman decided to unleash the full power of the wepons at his disposal. Despite having the knowledge and experience of Allied unstoppable power, something neither glamorous or sophisticated, they chose to mess about and play games. They no longer possessed any ability to participate in the debate as they had lost the battle. The Japanese had concerns regarding retention of their territorial integrity and respect for their emperor. The allies at that time stated that they now possessed a weapon of immense power and they would use it. They spelt out their terms which were unconditional surrender. The forces ranged against her were too strong. The war did not go the way the Japanese had hoped. War is terrible, and there is always a risk when you place your fate at the altar of the god of war. Japan decided to embark on a war of aggression. I haven't fully read all of this, but I will say my piece and then leave it at that. There was a lot of rhetoric from both vanquished nations (Japan and Germany in this context) about fighting to the last man.
If the Japanese had elected to continue to fight then the US would continue to inflict casualties until Japan conceded. To answer the question, the victor stops when the vanquished concedes defeat, exactly as happened between the US and her allies and Japan. When different military systems clash the result can be catastrophic. In any case Japan and the US were both most definitely 'modern' in this sense. The two terms are used in a specific way in this context. How much is enough varies across time and cultures, 20% casualties will convince most western militaries to hoist a white flag. You fight until such casualties on one side or the other that one of the two decides enough is enough and concedes. Modern warfare in the west dates back at least to the ancient Greeks. This is the essence of what military historians usually term 'modern' warfare as opposed to 'primitive' warfare.