A duty is a tariff or tax imposed on goods when transported across international borders. The principle purpose of any customs duty is to protect the country's economy, residents, jobs, environment, etc. by controlling the flow of goods, especially restrictive and prohibited goods, into and out of the country. Duties are typically imposed "ad valorem," meaning a percentage of the value of the goods being imported.
The primary difference between a tax and a fee is how the revenue is used. If the revenue is meant to raise money that can be used to defray the general costs of government it’s a tax. If the revenue is meant to pay for the costs of a specific government program or service it's a fee.
In summary, it's semantics. In addition to the duties, Customs and Border Protection collects a Merchandise Processing Fee and Harbor Maintenance Fee for most imported goods. By this definition, both fees are designed to pay for a specific government program (e.g., the maintenance of our harbors and port infrastructure). Similarly, the Alcohol and Tobacco Tax and Trade Bureau (TTB) collects an excise tax on imports of alcohol and tobacco. This money goes to a general fund or trust fund (along with gasoline, airline tickets, local telephone service, and other excise taxes) to pay for various goods and services.