Dynamic Programing

The traditional question of optimally deciding when to cut down a tree is among the most commonly posed questions asked of students learning the technique of dynamic programming. This paper shows that the traditional tree-cutting example is improperly formulated when the question of replanting is addressed, derives the proper method of finding optimal harvest length, and applies this method to an empirical forest growth function.

The traditional question of optimally deciding when to cut down a tree is among the most commonly posed questions asked of students learning the technique of dynamic programming. This paper shows that the traditional tree-cutting example is improperly formulated when the question of replanting is addressed, derives the proper method of finding optimal harvest length, and applies this method to an empirical forest growth function.

This paper presents the basics of Bellmann's dynamic programming, to be applied for radial fixed-track district heating networks. A decision-making model thereof is produced, and Garbai's scaling method [1] is applied in a new, narrower interval by using a tagging method well-known from computer science. Thereby the result yielded coincides with that of the previous method, but the number of calculations to be performed is reduced considerably.

We consider the problem of group testing with sum observations and noiseless answers, in which we aim to locate multiple objects by querying the number of objects in each of a sequence of chosen sets. We study a probabilistic setting with entropy loss, in which we assume a joint Bayesian prior density on the locations of the objects and seek to choose the sets queried to minimize the expected entropy of the Bayesian posterior distribution after a fixed number of questions. We present a new non-adaptive policy, called the dyadic policy, show it is optimal among nonadaptive policies, and is within a factor of two of optimal among adaptive policies. This policy is quick to compute, its nonadaptive nature makes it easy to parallelize, and our bounds show it performs well even when compared with adaptive policies. We also study an adaptive greedy policy, which maximizes the one-step expected reduction in entropy, and show that it performs at least as well as the dyadic policy, offering greater query efficiency but reduced parallelism. Numerical experiments demonstrate that both procedures outperform a divide-and-conquer benchmark policy from the literature, called sequential bifurcation, and show how these procedures may be applied in a stylized computer vision problem. Index Terms-Adaptive learning, Bayesian learning, computer vision, group testing, objects localization, optimal non-adaptive policy, rate of reduction in expected entropy. I. INTRODUCTION We consider the following set-guessing problem, which is similar to classical group testing [1], but differs in the form of the observations. We present the problem in one dimension for simplicity. An example in two dimensions is demonstrated in Section VII. Let Ω = R be the real line and θ = (θ 1 ,. .. , θ k) ∈ Ω k be a vector

We consider the problem of twenty questions with noiseless answers, in which we aim to locate multiple objects by querying the number of objects in each of a sequence of chosen sets. We assume a joint Bayesian prior density on the locations of the objects and seek to choose the sets queried to minimize the expected entropy of the Bayesian posterior distribution after a fixed number of questions. An optimal policy for accomplishing this task is characterized by the dynamic programming equations, but the curse of dimensionality prevents its tractable computation. We first derive a lower bound on the performance achievable by an optimal policy. We then provide explicit performance bounds relative to optimal for two computationally tractable policies: greedy, which maximizes the one-step expected reduction in entropy; and dyadic, which splits the search domain in successively finer partitions. We also show that greedy performs at least as well as the dyadic policy. This can help when choosing the policy most appropriate for a given application: the dyadic policy is easier to compute and nonadaptive, allowing its use in parallel settings or when questions are inexpensive relative to computation; while the greedy policy is more computationally intensive but also uses questions more efficiently, making it the better choice when robust sequential computation is possible. Numerical experiments demonstrate that both procedures outperform a divide-and-conquer benchmark policy from the literature, called sequential bifurcation. Finally, we further characterize performance under the dyadic policy by showing that the entropy of the posterior distribution is asymptotically normal.

This paper considers the task of finding a target location by making a limited number of sequential observation. Each observation results from evaluating an imperfect classifier of a chosen cost and accuracy on an interval of chosen length and position. Within a Bayesian framework, we study the problem of minimizing an objective that combines the entropy of the posterior distribution with the cost of the questions asked. In this problem, we show that the one-step lookahead policy is Bayes-optimal for any arbitrary time horizon. Moreover, this one-step lookahead policy is easy to compute and implement. We then use this policy in the context of localizing mitochondria in electron microscope images, and experimentally show that significant speed ups in acquisition can be gained, while maintaining near equal image quality at target locations, when compared to current policies.

We consider the problem of quickly localizing multiple targets by asking questions of the form "How many targets are within this set" while obtaining noisy answers. This setting is a generalization to multiple targets of the game of 20 questions in which only a single target is queried. We assume that the targets are points on the real line, or in a two dimensional plane for the experiments, drawn independently from a known distribution. We evaluate the performance of a policy using the expected entropy of the posterior distribution after a fixed number of questions with noisy answers. We derive a lower bound for the value of this problem and study a specific policy, named the dyadic policy. We show that this policy achieves a value which is no more than twice this lower bound when answers are noisefree, and show a more general constant factor approximation guarantee for the noisy setting. We present an empirical evaluation of this policy on simulated data for the problem of detecting multiple instances of the same object in an image. Finally, we present experiments on localizing multiple faces simultaneously on real images.

In a classic Markov decision problem of Derman, Lieberman, and Ross (1975) an investor has an initial capital x from which to make investments, the opportunities for which occur randomly over time. An investment of size y results in profit P (y), and the aim is maximize the sum of the profits obtained within a given time t. The problem is similar to a groundwater management problem of Burt (1964), the notorious bomber problem of Klinger and Brown (1968), and types of fighter problems addressed by Weber (1985), Shepp et al (1991) and Bartroff et al (2010a). In all these problems, one is allocating successive portions of a limited resource, optimally allocating y(x, t), as a function of remaining resource x and remaining time t. For their investment problem, Derman et al (1975) proved that an optimal policy has three monotonicity properties: (A) y(x, t) is nonincreasing in t, (B) y(x, t) is nondecreasing in x, and (C) x − y(x, t) is nondecreasing in x. Theirs is the only problem of its type for which all three properties are known to be true. In the bomber problem the status of (B) is unresolved. For the general fighter problem the status of (A) is unresolved. We survey what is known about these exceedingly difficult problems. We show that (A) and (C) remain true in the bomber problem, but that (B) is false if we very slightly relax the assumptions of the usual model. We give other new results, counterexamples and conjectures for these problems. Keywords Bomber problem • fighter problem • groundwater management problem • investment problem • Markov decision problem 1 Stochastic sequential allocation problems 1.1 An investment problem In a classic Markov decision problem posed by Derman, Lieberman, and Ross (1975) an investor has initial capital of x dollars, from which he can make withdrawals to fund investments. The opportunities for investment occur randomly as time proceeds. An investment of size y results in profit P (y), and the aim is maximize the sum of the profits obtained within a given time t. Without more ado let us set out this problem's dynamic programming equation. With an eye to variations of this problem to come, we make some notational changes. We take the remaining number of uninvested dollars to be discrete (a nonnegative integer) and denote it by n (= 0, 1,. . .) rather than x. The remaining time is also discrete and denoted by t = 0, 1,. .. . We suppose that an investment opportunity occurs with probability p (= 1 − q), independently at each remaining time step. We replace P (y) with a(k). Let F (n, t) denote the expected total payoff from investments made over t

In a classic Markov decision problem of Derman, Lieberman, and Ross (1975) an investor has an initial capital x from which to make investments, the opportunities for which occur randomly over time. An investment of size y results in profit P (y), and the aim is maximize the sum of the profits obtained within a given time t. The problem is similar to a groundwater management problem of Burt (1964), the notorious bomber problem of Klinger and Brown (1968), and types of fighter problems addressed by Weber (1985), Shepp et al (1991) and Bartroff et al (2010a). In all these problems, one is allocating successive portions of a limited resource, optimally allocating y(x, t), as a function of remaining resource x and remaining time t. For their investment problem, Derman et al (1975) proved that an optimal policy has three monotonicity properties: (A) y(x, t) is nonincreasing in t, (B) y(x, t) is nondecreasing in x, and (C) x − y(x, t) is nondecreasing in x. Theirs is the only problem of its type for which all three properties are known to be true. In the bomber problem the status of (B) is unresolved. For the general fighter problem the status of (A) is unresolved. We survey what is known about these exceedingly difficult problems. We show that (A) and (C) remain true in the bomber problem, but that (B) is false if we very slightly relax the assumptions of the usual model. We give other new results, counterexamples and conjectures for these problems. Keywords Bomber problem • fighter problem • groundwater management problem • investment problem • Markov decision problem 1 Stochastic sequential allocation problems 1.1 An investment problem In a classic Markov decision problem posed by Derman, Lieberman, and Ross (1975) an investor has initial capital of x dollars, from which he can make withdrawals to fund investments. The opportunities for investment occur randomly as time proceeds. An investment of size y results in profit P (y), and the aim is maximize the sum of the profits obtained within a given time t. Without more ado let us set out this problem's dynamic programming equation. With an eye to variations of this problem to come, we make some notational changes. We take the remaining number of uninvested dollars to be discrete (a nonnegative integer) and denote it by n (= 0, 1,. . .) rather than x. The remaining time is also discrete and denoted by t = 0, 1,. .. . We suppose that an investment opportunity occurs with probability p (= 1 − q), independently at each remaining time step. We replace P (y) with a(k). Let F (n, t) denote the expected total payoff from investments made over t

We consider a discrete-time groundwater model in which the cost of pumping takes a slightly different form to that which has been traditional in the research literature to date. This enables us to prove that (a) the optimal pumping quantity is nondecreasing in the ground water stock, (b) the stock level remaining after each period's pumping is also nondecreasing in the groundwater stock, (c) the optimal decision is determined by maximizing a concave function, and finally (d) the optimal pumping quantity is nonincreasing in the number of periods to go. We show that (a)-(c), while intuitive, do not hold under traditional modeling assumptions. We also explain the connections between our results and similar ones for some classic problems of operations research.

We consider a discrete-time groundwater model in which the cost of pumping takes a slightly different form to that which has been traditional in the research literature to date. This enables us to prove that (a) the optimal pumping quantity is nondecreasing in the ground water stock, (b) the stock level remaining after each period's pumping is also nondecreasing in the groundwater stock, (c) the optimal decision is determined by maximizing a concave function, and finally (d) the optimal pumping quantity is nonincreasing in the number of periods to go. We show that (a)-(c), while intuitive, do not hold under traditional modeling assumptions. We also explain the connections between our results and similar ones for some classic problems of operations research.

We consider a class of dynamic advertising problems under uncertainty in the presence of carryover and distributed forgetting effects, generalizing a classical model of Nerlove and Arrow. In particular, we allow the dynamics of the product goodwill to depend on its past values, as well as previous advertising levels. Building on previous work of two of the authors, the optimal advertising model is formulated as an infinite dimensional stochastic control problem. We obtain (partial) regularity as well as approximation results for the corresponding value function. Under specific structural assumptions we study the effects of delays on the value function and optimal strategy. In the absence of carryover effects, since the value function and the optimal advertising policy can be characterized in terms of the solution of the associated HJB equation, we obtain sharper characterizations of the optimal policy.

In today’s world, the development and continuation of life require energy. Supplying this energy demand requires careful and scientific planning of the energy provided by a variety of products, such as oil, gas, coal, electricity, etc. A new study on the operation of energy carriers called Energy Commitment (EC) is proposed. The purpose of the EC is to set a pattern for the use of energy carriers to supply energy demand, considering technical and economic constraints. EC is a constrained optimization problem that can be solved by using optimization methods. This study suggests the Following Optimization Algorithm (FOA) to solve the EC problem to achieve technical and economic benefits. Minimizing energy supply costs for the total study period is considered as an objective function. The FOA simulates social relationships among the community members who try to improve their community by following each other. Simulation is carried out on a 10-unit energy system supplied by various type...

We consider a class of dynamic advertising problems under uncertainty in the presence of carryover and distributed forgetting effects, generalizing the classical model of Nerlove and Arrow [33]. In particular, we allow the dynamics of the product goodwill to depend on its past values, as well as previous advertising levels. Building on previous work ([17]), the optimal advertising model is formulated as an infinite dimensional stochastic control problem. We obtain (partial) regularity as well as approximation results for the corresponding value function. Under specific structural assumptions we study the effects of delays on the value function and optimal strategy. In the absence of carryover effects, since the value function and the optimal advertising policy can be characterized in terms of the solution of the associated HJB equation, we obtain sharper characterizations of the optimal policy.

We study an optimal stopping problem for a stochastic differential equation with delay driven by a Lévy noise. Approaching the problem by its infinite-dimensional representation, we derive conditions yielding an explicit solution to the problem. Applications to the American put option problem are shown.

In today’s world, the development and continuation of life require energy. Supplying this energy demand requires careful and scientific planning of the energy provided by a variety of products, such as oil, gas, coal, electricity, etc. A new study on the operation of energy carriers called Energy Commitment (EC) is proposed. The purpose of the EC is to set a pattern for the use of energy carriers to supply energy demand, considering technical and economic constraints. EC is a constrained optimization problem that can be solved by using optimization methods. This study suggests the Following Optimization Algorithm (FOA) to solve the EC problem to achieve technical and economic benefits. Minimizing energy supply costs for the total study period is considered as an objective function. The FOA simulates social relationships among the community members who try to improve their community by following each other. Simulation is carried out on a 10-unit energy system supplied by various type...

In this paper we continue the research in Real Analysis, in the spirit of the very recent book [1]. Firstly, as a refinement of the global defect of integrability introduced in [1], § 5.1, we consider here a pointwise defect of integrability and study its properties and connections with the pointwise defect of continuity already introduced in [1], § 5.1. Secondly, as refinements of the global defects of monotonicity and of convexity introduced in the same book [1], § 5.2, we consider and study pointwise variants.

Purpose. In the present article, a new approach of the energy grid studies is introduced to program energy carriers. In this view, a proper plan is designed on the use of energy carriers considering the energy optimum use. Indeed, the proper energy grid is designed by applying Iran energy balance sheet information. It is proper to mention that, the energy grid modelling is done in a matrix form. The electrical energy distribution among power stations is achieved by using the particle swarm optimization algorithm. In the present paper, concerning the dynamic programming method, it is tried to determine a suitable combination of energy carriers. References 16, tables 17, figures 1.

This paper considers the task of finding a target location by making a limited number of sequential observations. Each observation results from evaluating an imperfect classifier of a chosen cost and accuracy on an interval of chosen length and position. Within a Bayesian framework, we study the problem of minimizing an objective that combines the entropy of the posterior distribution with the cost of the questions asked. In this problem, we show that the one-step lookahead policy is Bayes-optimal for any arbitrary time horizon. Moreover, this one-step lookahead policy is easy to compute and implement. We then use this policy in the context of localizing mitochondria in electron microscope images, and experimentally show that significant speed ups in acquisition can be gained, while maintaining near equal image quality at target locations, when compared to current policies.

We consider a class of dynamic advertising problems under uncertainty in the presence of carryover and distributed forgetting effects, generalizing the classical model of Nerlove and Arrow . In particular, we allow the dynamics of the product goodwill to depend on its past values, as well as previous advertising levels. Building on previous work ([16]), the optimal advertising model is formulated as an infinite dimensional stochastic control problem. We obtain (partial) regularity as well as approximation results for the corresponding value function. Under specific structural assumptions we study the effects of delays on the value function and optimal strategy. In the absence of carryover effects, since the value function and the optimal advertising policy can be characterized in terms of the solution of the associated HJB equation, we obtain sharper characterizations of the optimal policy.

In the paper recently proposed Human Factor Cepstral Coefficients (HFCC) are used to automatic recognition of pathological phoneme pronunciation in speech of impaired children and efficiency of this approach is compared to application of the standard Mel-Frequency Cepstral Coefficients (MFCC) as a feature vector. Both dynamic time warping (DTW), working on whole words or embedded phoneme patterns, and hidden Markov models (HMM) are used as classifiers in the presented research. Obtained results demonstrate superiority of combining HFCC features and modified phoneme-based DTW classifier.

We consider a class of dynamic advertising problems under uncertainty in the presence of carryover and distributed forgetting effects, generalizing the classical model of Nerlove and Arrow . In particular, we allow the dynamics of the product goodwill to depend on its past values, as well as previous advertising levels. Building on previous work ([16]), the optimal advertising model is formulated as an infinite dimensional stochastic control problem. We obtain (partial) regularity as well as approximation results for the corresponding value function. Under specific structural assumptions we study the effects of delays on the value function and optimal strategy. In the absence of carryover effects, since the value function and the optimal advertising policy can be characterized in terms of the solution of the associated HJB equation, we obtain sharper characterizations of the optimal policy.

We consider the problem of 20 questions with noisy answers, in which we seek to find a target by repeatedly choosing a set, asking an oracle whether the target lies in this set, and obtaining an answer corrupted by noise. Starting with a prior distribution on the target's location, we seek to minimize the expected entropy of the posterior distribution. We formulate this problem as a dynamic program and show that any policy optimizing the one-step expected reduction in entropy is also optimal over the full horizon. Two such Bayes-optimal policies are presented: one generalizes the probabilistic bisection policy due to Horstein and the other asks a deterministic set of questions. We study the structural properties of the latter, and illustrate its use in a computer vision application.

This paper considers the task of finding a target location by making a limited number of sequential observations. Each observation results from evaluating an imperfect classifier of a chosen cost and accuracy on an interval of chosen length and position. Within a Bayesian framework, we study the problem of minimizing an objective that combines the entropy of the posterior distribution with the cost of the questions asked. In this problem, we show that the one-step lookahead policy is Bayes-optimal for any arbitrary time horizon. ...

Abstract This paper considers the task of finding a target location by making a limited number of sequential observations. Each observation results from evaluating an imperfect classifier of a chosen cost and accuracy on an interval of chosen length and position. Within a Bayesian framework, we study the problem of minimizing an objective that combines the entropy of the posterior distribution with the cost of the questions asked.

We consider a discrete-time groundwater model in which the cost of pumping takes a slightly different form to that which has been traditional in the research literature to date. This enables us to prove that (a) the optimal pumping quantity is nondecreasing in the ground water stock, (b) the stock level remaining after each period's pumping is also nondecreasing in the groundwater stock, (c) the optimal decision is determined by maximizing a concave function, and finally (d) the optimal pumping quantity is nonincreasing in the number of periods to go. We show that (a)-(c), while intuitive, do not hold under traditional modeling assumptions. We also explain the connections between our results and similar ones for some classic problems of operations research.

Economic literature on groundwater management has traditionally been split into two areas: there are papers that evaluate different schemes of dynamic aquifer management, considering that pumping costs vary with stock but ignoring water quality. On the other hand, there are papers that consider contamination problems caused by specific pollutants. This paper presents two alternative models for joint quantity-quality management, and it shows that existing models are in fact special cases of these. The framework is dynamic and considers both the stock of water quantity and a stock measure of water quality. Optimal taxes are derived, and shown to be different from those in existing quantity-only or quality-only models. Implementation problems are briefly discussed.