We develop and test a general modeling framework to describe the sublethal effects of pollutants by adding toxicity modules to an established dynamic energy budget (DEB) model. The DEB model describes the rates of energy acquisition and expenditure by individual organisms; the toxicity modules describe how toxicants affect these rates by changing the value of one or more DEB parameters, notably the parameters quantifying the rates of feeding and maintenance. We investigate four toxicity modules that assume: (1) effects on feeding only; (2) effects on maintenance only; (3) effects on feeding and maintenance with similar values for the toxicity parameters; and (4) effects on feeding and maintenance with different values for the toxicity parameters. We test the toxicity modules by fitting each to published data on feeding, respiration, growth and reproduction. Among the pollutants tested are metals (mercury and copper) and various organic compounds (chlorophenols, toluene, polycyclic aromatic hydrocarbons, tetradifon and pyridine); organisms include mussels, oysters, earthworms, water fleas and zebrafish. In most cases, the data sets could be adequately described with any of the toxicity modules, and no single module gave superior fits to all data sets. We therefore propose that for many applications, it is reasonable to use the most general and parameter sparse module, i.e. module 3 that assumes similar effects on feeding and maintenance, as a default. For one example (water fleas), we use parameter estimates to calculate the impact of food availability and toxicant levels on the long term population growth rate.