Standing waves for a system of nonlinear Schrödinger equations in R N

Abstract

In this paper we study the existence of bound state solutions for stationary Schrödinger systems of the form $\begin{matrix} \{\begin{matrix} - Δ u + V (x) u = K (x) F_{u} (u, v) & in R^{N}, \\ - Δ v + V (x) v = K (x) F_{v} (u, v) & in R^{N}, \end{matrix} \end{matrix}$ where $N ⩾ 3$ , V and K are bounded continuous nonnegative functions, and $F (u, v)$ is a $C^{1}$ and p-homogeneous function with $2 < p < 2 N / (N - 2)$ . We give a special attention to the case when V may eventually vanishes. Our arguments are based on penalization techniques, variational methods and Moser iteration scheme.

Keywords

elliptic systems variational methods vanishing potential nonlinear Schrödinger equations bounded states

1. Introduction

In this paper, we study existence of solutions for the time-independent coupled nonlinear Schrödinger equations of the form $\begin{matrix} (S) & \{\begin{matrix} - Δ u + V (x) u = K (x) F_{u} (u, v) & in R^{N}, \\ - Δ v + V (x) v = K (x) F_{v} (u, v) & in R^{N}, \\ u > 0, v > 0 & in R^{N}, \\ u, & v \in H^{1} (R^{N}), \end{matrix} \end{matrix}$ where $N ⩾ 3$ , V and K are bounded continuous nonnegative functions, and $F : ([0, \infty) \times [0, \infty)) \to R$ is a p-homogeneous function of class $C^{1}$ with $2 < p < 2^{*}$ , where $2^{*} = 2 N / (N - 2)$ is the critical Sobolev exponent. Such class of systems arise in various branches of mathematical physics and they have been the subject of extensive study in recent years. Part of the interest is due to the fact that solutions of ( S ) are related to the existence of solitary wave solutions for nonlinear Schrödinger equations and Klein–Gordon equations (for a discussion see for example [8,14]). Indeed, $(u, v)$ is a solution of ( S ) if and only if $\begin{matrix} (ψ, φ), where ψ (x, t) = exp (i λ t) u (x) and φ (x, t) = exp (i λ t) v (x), \end{matrix}$ solves the nonlinear Schrödinger system $\begin{matrix} \{\begin{matrix} i \frac{\partial ψ}{\partial t} = - Δ ψ + \hat{V} (y) ψ - \frac{ψ}{| ψ |} K (x) F_{ψ} (| ψ |, | φ |) & in R^{N}, \\ i \frac{\partial φ}{\partial t} = - Δ ψ + \hat{V} (y) φ - \frac{φ}{| φ |} K (x) F_{φ} (| ψ |, | φ |) & in R^{N}, \end{matrix} \end{matrix}$ for $V (y) = \hat{V} (y) + λ$ . This class of systems has been studied recently due to its importance in various areas, for instance, physics. We are looking for solution $(u, v) \in H^{1} (R^{N}) \times H^{1} (R^{N})$ because standing waves u, v which have finite $L^{2}$ norm are the most relevant from the physical point of view since they correspond to bound states (cf. [1,3,10]).

Our work was motivated by some papers that have appeared in the recent years concerning the study of nonlinear Schrödinger equations by using purely variational approach since the seminal work [16]. We refer the reader to [1,2,5,6,10,12,17,18] and their bibliography for further studies. In order to apply variational arguments and to overcome the lack of compactness of the associated energy functional some authors have assumed that the potential is coercive and bounded away from zero. Here, in this paper our main purpose is to extend and complement the results in [4] to system ( S ) without any coercivity condition and with possible vanishing potential. This class of problems treated here has several difficulties. First, there is the usual lack of compactness of the Sobolev embedding, since our domain is the whole space $R^{N}$ . Second, since we are interested in vanishing potentials, it is challenging to find an adequate variational framework with an associated functional energy which critical points correspond to weak solutions of system ( S ). Although the approach is similar to that used in the scalar case, for the system case, in addition to the difficulty in handle the coupled terms, the truncation argument is completely different from the scalar case.

In the rest of this paper we will assume that $V, K : R^{N} \to R$ are bounded, nonnegative and continuous functions satisfying:

$\begin{matrix} λ_{1} : = inf_{(u, v) \in H, ‖ (u, v) ‖_{L} = 1} {‖ (u, v) ‖}_{H}^{2} > 0, \end{matrix}$ where $\begin{matrix} H : = {(u, v) \in H^{1} (R^{N}) \times H^{1} (R^{N}) : \int_{R^{N}} V (x) (u^{2} + v^{2}) d x < + \infty} \end{matrix}$ is a Hilbert space when endowed with the inner product $\begin{matrix} {⟨ (u, v), (ϕ, φ) ⟩}_{H} : = \int_{R^{N}} (\nabla u \nabla ϕ + V (x) u ϕ + \nabla v \nabla φ + V (x) v φ) d x, \forall (u, v), (ϕ, φ) \in H \end{matrix}$ and its correspondent norm $\begin{matrix} {‖ (u, v) ‖}_{H}^{2} : = \int_{R^{N}} (| \nabla u |^{2} + V (x) u^{2} + | \nabla v |^{2} + V (x) v^{2}) d x, \forall (u, v) \in H \end{matrix}$ and $L : = L^{2} (R^{N}) \times L^{2} (R^{N})$ equipped with the norm $‖ (u, v) ‖_{L}^{2} : = ‖ u ‖_{L^{2} (R^{N})}^{2} + ‖ v ‖_{L^{2} (R^{N})}^{2}$ (cf. [17]).

For the potential V and the function K, firstly, we assume that:

there exist $λ > 0$ and $R > 0$ , such that $\begin{matrix} \{\begin{matrix} \exists x_{o} \in B_{R} (0) such that K (x_{o}) > 0 and \\ 0 < μ ⩽ K (x) ⩽ V (x) ⩽ λ < k_{p} : = \frac{2 p}{p - 2}, \forall | x | ⩾ R . \end{matrix} \end{matrix}$

We also impose for K, a similar hyphotesis used in [4], namely,

there exist $γ > μ$ and $R > 0$ , such that $\begin{matrix} sup_{| x | ⩾ R} \frac{R^{2 (N - 2)}}{| x |^{2 (N - 2)}} K (x) ⩽ γ . \end{matrix}$

In order to state our main result let us introduce the assumptions on the p-homogeneous function F that we assume throughout this article:

there exists $0 < c_{0} ⩽ μ^{p / 2}$ such that $\begin{matrix} | F_{u} (u, v) | + | F_{v} (u, v) | ⩽ c_{0} (u^{p - 1} + v^{p - 1}), \forall u, v ⩾ 0 . \end{matrix}$

$F_{u} (0, 1) = F_{v} (1, 0) = 0$ .

$F_{u} (1, 0) = F_{v} (0, 1) = 0$ .

$F_{u v} (u, v) > 0$ , $\forall u, v > 0$ .

Throughout this paper a positive solution $(u, v)$ of ( S ) means that $u > 0$ and $v > 0$ in $R^{N}$ . We now state the result which will be proved about the existence of solutions of system ( S ).

Theorem 1.1.
Suppose that $(V_{0})$ and $(F_{0})$ – $(F_{3})$ are satisfied. Then, there exists $γ^{} > 0$ such that* ( S ) has a positive weak solution for any potentials that satisfy $(V_{1})$ – $(V_{2})$ with $γ ⩽ γ^{*}$ .
Remark 1.1.

A typical example of p-homogeneous function F satisfying our hypotheses is given by $F (u, v) = Q {(u, v)}^{p / l}$ with $p ⩾ l$ and $Q (u, v)$ is a l-homogeneous function satisfying $\begin{matrix} Q (u, v) = \sum_{α_{i} + β_{i} = l} a_{i} u^{α_{i}} v^{β_{i}}, u, v ⩾ 0, \end{matrix}$ where $i \in I$ ( $# I < \infty$ ), $α_{i} ⩾ 2$ , $β_{i} ⩾ 2$ , $N = 3$ , $4 ⩽ l ⩽ p < 2 N / (N - 2)$ and $a_{i} > 0$ .

For some $R > 0$ let $V, K$ be bounded, nonnegative and continuous functions which are constants for all $| x | ⩽ R$ and such that $0 < μ ⩽ K (x) ⩽ V (x) ⩽ λ < k_{p}$ for all $| x | ⩾ R$ . It is easy to see that $V, K$ satisfy assumptions $(V_{0})$ – $(V_{2})$ .

This paper is organized as follows. In Section 2, we give some preliminaries and we introduce an auxiliary system ( AS ) appropriated to apply the minimax method. In Section 3, from condition $(V_{1})$ , we prove the Palais–Smale sequence compactness condition for the associated functional to the auxiliary system ( AS ) and with the help of the mountain-pass theorem get the existence of critical points. In Section 4, we studied some qualitative properties of the solutions of ( AS ), more precisely, we proved that the solutions of ( AS ) belongs to $L^{\infty} (R^{N})$ and we also obtain the decay rate to zero at infinity of these solutions. In Section 5, using the condition $(V_{2})$ , we get a positive solution for the auxiliary system ( AS ) which is also a solution for the system ( S ) and therefore we conclude the proof of the Theorem 1.1.
2. Variational framework

2.1. The auxiliary system

We are looking for solutions of ( S ) defined in $R^{N}$ . To overcome the lack of compactness, we will use the method introduced in [11]. For that, we formulate our problem in the weighted Sobolev space H and we introduce an auxiliary system modifying the gradient $(F_{u} (u, v), F_{v} (u, v))$ for a $C^{1}$ gradient, for which we can guarantee that Palais–Smale sequences for the associated functional of the auxiliary system has a strongly convergent subsequence in H. Since here we are interested in the existence of positive solutions of ( S ) in the sense that each coordinate is a positive function, we redefine the nonlinearity $F (t, s)$ as $F (t, s) = 0$ if $t ⩽ 0$ or $s ⩽ 0$ .

Remark 2.1.

Using condition $(F_{0})$ , we have $\begin{matrix} (2.1) & | p F (u, v) | ⩽ 3 c_{0} (u^{p} + v^{p}), \forall u, v ⩾ 0 . \end{matrix}$ Moreover, $F_{u} (u, v), F_{v} (u, v) \in L^{p / (p - 1)} (R^{N})$ for all $u, v \in L^{p} (R^{N})$ .

By assumptions $(F_{1})$ – $(F_{2})$ , we deduce that $F_{u} (u, v) = F_{v} (u, v) = 0$ if either $u = 0$ or $v = 0$ . In addition, by the homogeneity of F, $\begin{matrix} p F (u, v) = u F_{u} (u, v) + v F_{v} (u, v), for all u, v ⩾ 0, \end{matrix}$ and thus, $F (u, 0) = F (0, v) = 0$ , for all $u, v ⩾ 0$ .

From the hypotheses $(F_{2})$ – $(F_{3})$ , we see that $F_{u} (u, v)$ , $F_{v} (u, v)$ are nonnegative functions, and thus thanks to the homogeneity property, the same holds for $F (u, v)$ .

By assumption $(V_{1})$ , $Z = {x \in R^{N} : V (x) = 0}$ is a compact set. We recall that $Z$ can be empty set.

Using the definition of weighted Sobolev space H and the Sobolev embedding theorem, the following embedding are continuous by condition $(V_{0})$ : $\begin{array}{rcl} H & ↪ & H^{1} (R^{N}) \times H^{1} (R^{N}) ↪ L^{p} (R^{N}) \times L^{p} (R^{N}), 2 ⩽ p ⩽ 2^{}, N ⩾ 3 . \end{array}$ Let I denote the Euler–Lagrange functional associated with system ( S ) given by $\begin{matrix} I (u, v) = \frac{1}{2} {‖ (u, v) ‖}_{H}^{2} - \int_{R^{N}} K (x) F (u, v) d x \end{matrix}$ defined on the Hilbert space H. It is well known that $I \in C^{1} (H, R)$ with Gateaux derivative given by $\begin{matrix} I^{'} (u, v) (w, z) = {⟨ (u, v), (w, z) ⟩}_{H} - \int_{R^{N}} K (x) (F_{u} (u, v) w + F_{v} (u, v) z) d x . \end{matrix}$ It is standard to prove that critical points of I corresponds to the weak solutions of ( S ) (cf. [15]).

Let R given in $(V_{1})$ , $Λ = B_{R} (0) \subset R^{N}$ and $k = k_{p} = 2 p / (p - 2) > 2$ . Let $a > 0$ be a real constant which will be chosen appropriately. Let us consider the cutoff function $\begin{matrix} (2.2) & \{\begin{matrix} η : R \to R & decreasing and C^{1}, \\ η (t) = 1, & \forall t \in (- \infty, a], \\ η (t) = 0, & \forall t \in [5 a, + \infty), \\ | η^{'} (t) | ⩽ 1 / (5 a), & \forall t \in R . \end{matrix} \end{matrix}$ Now, consider $\hat{F} : (R^{N} ∖ Λ) \times R \times R \to R$ given by $\begin{matrix} \hat{F} (x, t, s) : = η (| {(K (x))}^{1 / 2} (t, s) |) F (t, s) + (1 - η (| {(K (x))}^{1 / 2} (t, s) |)) A (x) \frac{K (x)}{2 k} (t^{2} + s^{2}), \end{matrix}$ where $\begin{matrix} A (x) : = max {\frac{2 k F (t, s)}{K (x) (t^{2} + s^{2})} : (s, t) \in R^{2} and a ⩽ \sqrt{K (x) (t^{2} + s^{2})} ⩽ 5 a} . \end{matrix}$ Thus, from (2.1), $A (x) \to 0$ uniformly as $a \to 0$ . Note that $\hat{F}$ is well defined, nonnegative and is of class $C^{1}$ (cf. item 3, Remark 2.1). We can now define $G : R^{N} \times R \times R \to R$ given by $\begin{matrix} G (x, t, s) : = χ_{Λ} (x) F (t, s) + (1 - χ_{Λ} (x)) \hat{F} (x, t, s), \end{matrix}$ where $χ_{Λ}$ denotes the characteristic function of the set Λ. Thus, G is a p-homogeneous function on Λ and $G (x, t, s) ⩾ 0$ , $\forall (x, t, s) \in R^{N} \times R \times R$ . Moreover, for fixed $x \in R^{N}$ the function $(t, s) \mapsto G (x, t, s)$ is of class $C^{1}$ and for each fixed $(t, s) \in R^{2}$ the function $x \mapsto G (x, s, t)$ is Lebesgue measurable in $R^{N}$ . We can now introduce the auxiliary system $\begin{array}{l} (AS) & \{\begin{matrix} - Δ u + V (x) u = K (x) G_{u} (x, u, v) & in R^{N}, \\ - Δ v + V (x) v = K (x) G_{v} (x, u, v) & in R^{N}, \\ u > 0, v > 0 & in R^{N}, \\ u, & v \in H^{1} (R^{N}) . \end{matrix} \end{array}$ The Euler–Lagrange functional associated with ( AS ), $J : H \to R$ is given by $\begin{array}{rcl} J (u, v) & = & \frac{1}{2} {‖ (u, v) ‖}_{H}^{2} - \int_{R^{N}} K (x) G (x, u, v) d x . \end{array}$ From the conditions on G, the functional J belongs to class $C^{1} (H, R)$ (cf. [15]) and its Gateaux derivative is given by $\begin{matrix} J^{'} (u, v) (w, z) = {⟨ (u, v), (w, z) ⟩}_{H} - \int_{R^{N}} K (x) (G_{u} (x, u, v) w + G_{v} (x, u, v) z) d x, \end{matrix}$ for any $(u, v), (w, z) \in H$ . Thus, critical points of functional J correspond to weak solutions of system ( AS ). Lemma 2.1.
The function G satisfies the following properties:* $\begin{matrix} (2.3) & p G (x, u, v) = u G_{u} (x, u, v) + v G_{v} (x, u, v), \forall x \in Λ . \end{matrix}$ Moreover, for $k = 2 p / (p - 2)$ , we can choose the constant a sufficiently small such that $\begin{matrix} (2.4) & 2 G (x, u, v) ⩽ u G_{u} (x, u, v) + v G_{v} (x, u, v) ⩽ \frac{K (x)}{k} (u^{2} + v^{2}), \forall x \in R^{N} ∖ Λ . \end{matrix}$
Proof.
Using the definition of G, we have that $G (x, u, v) = F (u, v)$ for all $x \in Λ$ , and consequently (2.3) holds.

Observe that $G (x, u, v) = \hat{F} (x, u, v)$ for all $x \in R^{N} ∖ Λ$ , besides that from the definition of the function $\hat{F}$ , we have for all $x \in R^{N} ∖ Λ$ $\begin{array}{rcl} {\hat{F}}_{u} & = & η (| K^{1 / 2} (u, v) |) F_{u} (u, v) + \frac{K}{k} u A (x) (1 - η (| K^{1 / 2} (u, v) |)) \\ (2.5) & + η^{'} (| K^{1 / 2} (u, v) |) (K^{1 / 2} u {(u^{2} + v^{2})}^{- 1 / 2}) [F (u, v) - A (x) (\frac{K (x)}{2 k} (u^{2} + v^{2}))] \end{array}$ and $\begin{array}{rcl} {\hat{F}}_{v} & = & η (| K^{1 / 2} (u, v) |) F_{v} (u, v) + \frac{K}{k} v A (x) (1 - η (| K^{1 / 2} (u, v) |)) \\ (2.6) & + η^{'} (| K^{1 / 2} (u, v) |) (K^{1 / 2} v {(u^{2} + v^{2})}^{- 1 / 2}) [F (u, v) - A (x) (\frac{K}{2 k} (u^{2} + v^{2}))] . \end{array}$ They imply that $\begin{array}{rcl} u {\hat{F}}_{u} + v {\hat{F}}_{v} & = & p η (| K^{1 / 2} (u, v) |) F (u, v) + A (x) [\frac{K}{k} (u^{2} + v^{2})] (1 - η (| K^{1 / 2} (u, v) |)) \\ (2.7) & + η^{'} (| K^{1 / 2} (u, v) |) (K^{1 / 2} {(u^{2} + v^{2})}^{1 / 2}) [F (u, v) - A (x) (\frac{K}{2 k} (u^{2} + v^{2}))] . \end{array}$

Therefore, $\begin{array}{rcl} u {\hat{F}}_{u} + v {\hat{F}}_{v} & ⩾ & p η (| K^{1 / 2} (u, v) |) F (u, v) + A (x) [\frac{K}{k} (u^{2} + v^{2})] (1 - η (| K^{1 / 2} (u, v) |)) \\ ⩾ & 2 \hat{F} (x, u, v) for all x \in R^{N} ∖ Λ . \end{array}$ Observe that $\begin{matrix} \frac{p F (u, v)}{\frac{K}{2 k} (u^{2} + v^{2})} ⩽ 3 c_{0} \frac{| u |^{p} + | v |^{p}}{\frac{K}{2 k} (u^{2} + v^{2})} ⩽ \frac{5^{p - 2} 6 k c_{0} a^{p - 2}}{μ^{p / 2}} ⩽ C a^{p - 2} . \end{matrix}$ Where $(u, v) \in R^{2}$ such that $a ⩽ \sqrt{K (x) (u^{2} + v^{2})} ⩽ 5 a$ and $C > 0$ is a real constant. Since $p > 2$ then, we can conclude that taking a sufficiently small, we get $\begin{matrix} \frac{p F (u, v)}{\frac{K (x)}{2 k} (u^{2} + v^{2})} ⩽ 1 . \end{matrix}$ Beside that we recall that $A (x) ⩾ 0$ and $A (x) \to 0$ uniformly as $a \to 0$ for all $x \in R^{N} ∖ Λ$ . From (2.7) $\begin{array}{rcl} \frac{u {\hat{F}}_{u} + v {\hat{F}}_{v}}{\frac{K}{2 k} (u^{2} + v^{2})} & = & p η (| K^{1 / 2} (u, v) |) \frac{F (u, v)}{\frac{K}{2 k} (u^{2} + v^{2})} + 2 A (x) (1 - η (| K^{1 / 2} (u, v) |)) \\ + η^{'} (| K^{1 / 2} (u, v) |) (K^{1 / 2} {(u^{2} + v^{2})}^{1 / 2}) [\frac{F (u, v)}{\frac{K}{2 k} (u^{2} + v^{2})} - A (x)] . \end{array}$

Therefore, $\begin{matrix} \frac{u {\hat{F}}_{u} + v {\hat{F}}_{v}}{\frac{K (x)}{2 k} (u^{2} + v^{2})} ⩽ 2 . \end{matrix}$ That is, $\begin{matrix} u {\hat{F}}_{u} + v {\hat{F}}_{v} ⩽ \frac{K (x)}{k} (u^{2} + v^{2}) for all x \in R^{N} ∖ Λ . \end{matrix}$ □

2.2. Mountain-pass geometry

It is standard to prove that J satisfies the mountain pass geometry, we include the proof for ready reference. See [15,16] for more details.

Lemma 2.2 (Mountain–pass geometry).

The functional J has the mountain pass geometry, that is, J satisfies:

there is $ρ, α > 0$ , such that $J (u, v) ⩾ α$ if $‖ (u, v) ‖_{H} = ρ$ ,

for any $(u, v) \in H$ , $u, v > 0$ with compact support on Λ, there is $0 < θ < 1$ such that $J (t^{θ} u, t^{θ} v) \to - \infty$ as $t \to + \infty$ .

Proof.
From Remark 2.1 we have that $\begin{matrix} | p G (x, u, v) | ⩽ 3 c_{0} (| u |^{p} + | v |^{p}) for all x \in Λ . \end{matrix}$ By the Sobolev embedding $\begin{matrix} \int_{Λ} K (x) G (x, u, v) ⩽ \int_{R^{N}} K (x) F (u, v) ⩽ C_{1} {‖ (u, v) ‖}_{H}^{p}, \end{matrix}$ where $C_{1}$ is a positive constant. Using the estimative above, we have that $\begin{array}{rcl} J (u, v) & = & \frac{1}{2} {‖ (u, v) ‖}_{H}^{2} - \int_{Λ} K (x) G (x, u, v) d x - \int_{R^{N} ∖ Λ} K (x) G (x, u, v) d x \\ ⩾ & \frac{1}{2} {‖ (u, v) ‖}_{H}^{2} - C_{1} {‖ (u, v) ‖}_{H}^{p} - \int_{R^{N} ∖ Λ} λ \frac{V (x)}{2 k} (u^{2} + v^{2}) d x \\ ⩾ & ((\frac{1}{2} - \frac{λ}{2 k}) - C_{1} {‖ (u, v) ‖}_{H}^{p - 2}) {‖ (u, v) ‖}_{H}^{2} . \end{array}$ Choose $ρ > 0$ such that $D = (\frac{1}{2} - \frac{λ}{2 k} - C_{1} ‖ (u, v) ‖_{H}^{p - 2}) > 0$ . Let $α = D ρ^{2}$ . Then exists $ρ, α > 0$ such that $J (u, v) ⩾ α$ , for all $(u, v) \in \partial B (0, ρ)$ .

Now, we will prove (2). Consider $θ > 0$ such that $1 / p < θ < 1 / 2$ . From the definition of G, $G (x, u, v) = F (u, v)$ for $x \in Λ$ . So, $\begin{matrix} p G (x, u, v) = u G_{u} (x, u, v) + v G_{v} (x, u, v) where x \in Λ, \end{matrix}$ therefore we have $\begin{array}{rcl} \frac{d}{d t} {G (x, t^{θ} u, t^{θ} v)} & = & p θ \frac{1}{t} G (x, t^{θ} u, t^{θ} v) \\ (2.8) & ⩾ & \frac{1}{t} G (x, t^{θ} u, t^{θ} v) . \end{array}$ Observe that $p θ > 1$ and from $(F_{2})$ – $(F_{3})$ and the definition of G, we have $G (x, u, v) = F (u, v) > 0$ where $x \in Λ$ (cf. Remark 2.1(3)).

The inequality (2.8) implies that $G (x, t^{θ} u, t^{θ} v) ⩾ t Q (x, u, v)$ for some function Q. Then, $\begin{matrix} J (t^{θ} u, t^{θ} v) ⩽ \frac{1}{2} t^{2 θ} {‖ (u, v) ‖}_{H}^{2} - \int_{Λ} t K (x) Q (x, u, v) . \end{matrix}$ Since $2 θ < 1$ , we conclude that for any $(u, v) \in H ∖ (0, 0)$ fixed, $u, v > 0$ with compact support on Λ, we have $\begin{matrix} J (t^{θ} u, t^{θ} v) \to - \infty as t \to + \infty . \end{matrix}$ This completes the proof. □

3. The Palais–Smale condition

In order to apply critical point theory to prove the existence of weak solutions of ( AS ), we first need to study some compactness property of functional J.

Lemma 3.1.
Suppose that $(F_{0})$ – $(F_{3})$ hold. Then, any Palais–Smale sequence for J is bounded in H.
Proof.
Let $(u_{n}, v_{n}) \subset H$ be a Palais–Smale sequence for J, that is, $\begin{matrix} | J (u_{n}, v_{n}) | ⩽ c and J^{'} (u_{n}, v_{n}) \to 0 as n \to \infty in H^{'} . \end{matrix}$ Thus, $\begin{matrix} {‖ (u_{n}, v_{n}) ‖}_{H}^{2} = \int_{R^{N}} K (x) (G_{u} (u_{n}, v_{n}) u_{n} + G_{v} (u_{n}, v_{n}) v_{n}) d x + o ({‖ (u_{n}, v_{n}) ‖}_{H}) \end{matrix}$ and $\begin{matrix} \frac{1}{2} {‖ (u_{n}, v_{n}) ‖}_{H}^{2} = \int_{R^{N}} K (x) G (x, u_{n}, v_{n}) d x + O (1) . \end{matrix}$ Consider $Ω = R^{N} ∖ Λ$ . Recalling that $k = k_{p} = 2 p / (p - 2)$ , we have $\begin{array}{rcl} (\frac{1}{2} - \frac{1}{p}) {‖ (u_{n}, v_{n}) ‖}_{H}^{2} & ⩽ & \int_{Λ} K (x) (G (x, u_{n}, v_{n}) - \frac{1}{p} [u_{n} G_{u} (x, u_{n}, v_{n}) + v_{n} G_{v} (x, u_{n}, v_{n})]) d x \\ + \int_{Ω} K (x) (G (x, u_{n}, v_{n}) - \frac{1}{p} [u_{n} G_{u} (x, u_{n}, v_{n}) + v_{n} G_{v} (x, u_{n}, v_{n})]) d x \\ + O (1) + o ({‖ (u_{n}, v_{n}) ‖}_{H}) . \end{array}$

Using (2.3), (2.4), the condition $(V_{1})$ , we deduced that $\begin{matrix} (3.1) & (\frac{1}{2} - \frac{1}{p}) {‖ (u_{n}, v_{n}) ‖}_{H}^{2} ⩽ \int_{Ω} \frac{λ V (x)}{k^{2}} u_{n}^{2} + \frac{λ V (x)}{k^{2}} v_{n}^{2} d x + O (1) + o ({‖ (u_{n}, v_{n}) ‖}_{H}), \end{matrix}$ which implies that $\begin{matrix} (\frac{1}{k} - \frac{λ}{k^{2}}) {‖ (u_{n}, v_{n}) ‖}_{H}^{2} ⩽ O (1) + o ({‖ (u_{n}, v_{n}) ‖}_{H}) . \end{matrix}$ Therefore, $(u_{n}, v_{n})$ is bounded in H. □
Lemma 3.2.
Suppose that and $(F_{0})$ – $(F_{3})$ hold. Let $(u_{n}, v_{n}) \subset H$ be an arbitrary Palais–Smale sequence of J. Then, J satisfies the Palais–Smale condition, that is, there is a $(u_{n_{m}}, v_{n_{m}}) \subset H$ subsequence of $(u_{n}, v_{n})$ such that $(u_{n_{m}}, v_{n_{m}}) \to (u, v)$ in H, as $m \to \infty$ .
Proof.
From Lemma 3.1, going if necessary to a subsequence, we can assume that there exists $(u, v) \in H$ such that $(u_{n}, v_{n}) ⇀ (u, v)$ weakly in H, as $n \to \infty$ . Observe that $G_{u} (x, u, v), G_{v} (x, u, v) \in L^{p / (p - 1)} (R^{N})$ , therefore we have $\begin{matrix} (3.2) & \int_{R^{N}} G_{u} (x, u, v) (u_{n} - u) d x \to 0 and \int_{R^{N}} G_{v} (x, u, v) (v_{n} - v) d x \to 0, \end{matrix}$ as $n \to \infty$ . For each $ε > 0$ , let $r > R$ be such that $\begin{matrix} (3.3) & \int_{R^{N} ∖ B_{r}} G_{u} (x, u_{n}, v_{n}) u d x < \frac{ε}{3}, \int_{R^{N} ∖ B_{r}} G_{v} (x, u_{n}, v_{n}) v d x < \frac{ε}{3} \end{matrix}$ and $\begin{matrix} (3.4) & 2 {(1 - \frac{λ}{k})}^{- 1} ω_{N}^{1 / N} C {(\int_{r ⩽ | x | ⩽ 2 r} max {(| u (x) |, | v (x) |)}^{2^{}} d x)}^{1 / 2^{}} < \frac{ε}{6}, \end{matrix}$ where $C ⩾ ‖ (u_{n}, v_{n}) ‖_{H}$ is a positive constant and $ω_{N}$ is the volume of the unitary ball in $R^{N}$ . Let $ξ = ξ_{r} \in C_{0}^{\infty} (R^{N}, [0, 1])$ , be a function verifying $supp ξ \subseteq B_{r}^{c} (0)$ , $ξ \equiv 1$ in $B_{2 r}^{c} (0)$ , $0 < ξ < 1$ if $r < | x | < 2 r$ and $\begin{matrix} | ξ^{'} (x) | ⩽ \frac{1}{r}, for all x \in R^{N} . \end{matrix}$ Since $(u_{n}, v_{n})$ is bounded in H and $| ξ (x) | ⩽ 1$ , the sequence $(ξ (u_{n}, v_{n}))$ is also bounded, and then $J^{'} (u_{n}, v_{n}) \cdot (ξ (u_{n}, v_{n})) = o_{n} (1)$ , that is, $\begin{array}{rcl} \int_{R^{N}} (\nabla u_{n} \nabla (ξ u_{n}) + \nabla v_{n} \nabla (ξ v_{n})) d x + \int_{R^{N}} V (x) u_{n} (ξ u_{n}) + V (x) v_{n} (ξ v_{n}) d x \\ = \int_{R^{N}} K (x) ((ξ u_{n}) G_{u} (x, u_{n}, v_{n}) + (ξ v_{n}) G_{v} (x, u_{n}, v_{n})) d x + o_{n} (1) . \end{array}$ Since $ξ \equiv 0$ in $B_{r} (0)$ and $Λ = B_{R} (0)$ , the last equality combined with the property (2.4) and condition $(V_{1})$ yields $\begin{array}{rcl} \int_{| x | ⩾ r} (| \nabla u_{n} |^{2} + V (x) u_{n}^{2} + | \nabla v_{n} |^{2} + V (x) v_{n}^{2}) ξ d x \\ ⩽ \frac{λ}{k} \int_{| x | ⩾ r} (ξ V (x) u_{n}^{2} + ξ V (x) v_{n}^{2}) d x - \int_{| x | ⩾ r} (u_{n} \nabla u_{n} \nabla ξ + v_{n} \nabla v_{n} \nabla ξ) d x + o_{n} (1) \end{array}$ and so, $\begin{array}{rcl} (1 - \frac{λ}{k}) \int_{| x | ⩾ 2 r} (| \nabla u_{n} |^{2} + V (x) u_{n}^{2} + | \nabla v_{n} |^{2} + V (x) v_{n}^{2}) d x \\ (3.5) & ⩽ \frac{1}{r} \int_{r ⩽ | x | ⩽ 2 r} (| u_{n} | | \nabla u_{n} | + | v_{n} | | \nabla v_{n} |) d x + o_{n} (1) . \end{array}$ Here we have used assumption $(V_{1})$ to guarantee that $λ < k_{p}$ . By Holder’s inequality, $\begin{matrix} \{\begin{matrix} \int_{r ⩽ | x | ⩽ 2 r} | u_{n} | | \nabla u_{n} | d x ⩽ ‖ \nabla u_{n} ‖_{L^{2} (R^{N})} {(\int_{r ⩽ | x | ⩽ 2 r} | u_{n} |^{2} d x)}^{1 / 2}, \\ \int_{r ⩽ | x | ⩽ 2 r} | v_{n} | | \nabla v_{n} | d x ⩽ ‖ \nabla v_{n} ‖_{L^{2} (R^{N})} {(\int_{r ⩽ | x | ⩽ 2 r} | v_{n} |^{2} d x)}^{1 / 2} . \end{matrix} \end{matrix}$ Due to the Rellich–Kondrachov compactness theorem, we have that $(u_{n}, v_{n}) \to (u, v)$ as $n \to \infty$ in $L^{2} (B_{2 r} ∖ B_{r})$ and using that $(u_{n}, v_{n})$ is bounded, it follows that $\begin{matrix} (3.6) & \{\begin{matrix} {lim sup}_{n} \int_{r ⩽ | x | ⩽ 2 r} | u_{n} | | \nabla u_{n} | d x ⩽ C {(\int_{r ⩽ | x | ⩽ 2 r} | u |^{2} d x)}^{1 / 2}, \\ {lim sup}_{n} \int_{r ⩽ | x | ⩽ 2 r} | v_{n} | | \nabla v_{n} | d x ⩽ C {(\int_{r ⩽ | x | ⩽ 2 r} | v |^{2} d x)}^{1 / 2} . \end{matrix} \end{matrix}$ On the other hand, using again Holder’s inequality $\begin{matrix} (3.7) & \{\begin{matrix} {(\int_{r ⩽ | x | ⩽ 2 r} | u |^{2} d x)}^{1 / 2} ⩽ {(\int_{r ⩽ | x | ⩽ 2 r} | u |^{2^{}} d x)}^{1 / 2^{}} | B_{2 r} ∖ B_{r} |^{1 / N}, \\ {(\int_{r ⩽ | x | ⩽ 2 r} | v |^{2} d x)}^{1 / 2} ⩽ {(\int_{r ⩽ | x | ⩽ 2 r} | v |^{2^{}} d x)}^{1 / 2^{}} | B_{2 r} ∖ B_{r} |^{1 / N} . \end{matrix} \end{matrix}$ Recalling that $| B_{2 r} ∖ B_{r} | ⩽ | B_{2 r} | = ω_{N} 2^{N} r^{N}$ , from (3.6) and (3.7) $\begin{matrix} (3.8) & \{\begin{matrix} {lim sup}_{n} \int_{r ⩽ | x | ⩽ 2 r} | u_{n} | | \nabla u_{n} | d x ⩽ 2 ω_{N}^{1 / N} r C {(\int_{r ⩽ | x | ⩽ 2 r} | u |^{2^{}} d x)}^{1 / 2^{}}, \\ {lim sup}_{n} \int_{r ⩽ | x | ⩽ 2 r} | v_{n} | | \nabla v_{n} | d x ⩽ 2 ω_{N}^{1 / N} r C {(\int_{r ⩽ | x | ⩽ 2 r} | v |^{2^{}} d x)}^{1 / 2^{}} . \end{matrix} \end{matrix}$ By choosing $r > 0$ in (3.4), (3.5) and (3.8), it implies that $\begin{matrix} (3.9) & 0 ⩽ \underset{n}{lim sup} \int_{| x | ⩾ 2 r} (| \nabla u_{n} |^{2} + V (x) u_{n}^{2} + | \nabla v_{n} |^{2} + V (x) v_{n}^{2}) d x < \frac{ε}{3} . \end{matrix}$ Now, using the Sobolev compact embedding and dominated convergence theorem, they lead to $\begin{array}{rcl} lim_{n \to \infty} \int_{| x | ⩽ 2 r} K (x) (u_{n} G_{u} (x, u_{n}, v_{n}) + v_{n} G_{v} (x, u_{n}, v_{n})) d x \\ (3.10) & = \int_{| x | ⩽ 2 r} K (x) (u G_{u} (x, u, v) + v G_{v} (x, u, v)) d x \end{array}$ and $\begin{array}{rcl} lim_{n \to \infty} \int_{| x | ⩽ 2 r} K (x) (u G_{u} (x, u_{n}, v_{n}) + v G_{v} (x, u_{n}, v_{n})) d x \\ (3.11) & = \int_{| x | ⩽ 2 r} K (x) (u G_{u} (x, u, v) + v G_{v} (x, u, v)) d x . \end{array}$ From (3.9) and (3.10) we have that $\begin{array}{rcl} lim_{n \to \infty} \int_{R^{N}} K (x) (u_{n} G_{u} (x, u_{n}, v_{n}) + v_{n} G_{v} (x, u_{n}, v_{n})) d x \\ (3.12) & = \int_{R^{N}} K (x) (u G_{u} (x, u, v) + v G_{v} (x, u, v)) d x . \end{array}$ Finally, from (3.3) and (3.11), we get $\begin{array}{rcl} lim_{n \to \infty} \int_{R^{N}} K (x) (u G_{u} (x, u_{n}, v_{n}) + v G_{v} (x, u_{n}, v_{n})) d x \\ (3.13) & = \int_{R^{N}} (u G_{u} (x, u, v) + v G_{v} (x, u, v)) d x . \end{array}$ Observing that $\begin{array}{rcl} {‖ (u_{n}, v_{n}) - (u, v) ‖}_{H}^{2} & = & ⟨ J^{'} (u_{n}, v_{n}) - J^{'} (u, v), (u_{n} - u, v_{n} - v) ⟩ \\ + \int_{R^{N}} K (x) (G_{u} (x, u_{n}, v_{n}) - G_{u} (x, u, v)) (u_{n} - u) d x \\ + \int_{R^{N}} K (x) (G_{v} (x, u_{n}, v_{n}) - G_{v} (x, u, v)) (v_{n} - v) d x . \end{array}$ Using (3.2), (3.12) and (3.13) together with $J^{'} (u_{n}, v_{n}) (u_{n}, v_{n}) = o_{n} (1)$ , yield $‖ (u_{n}, v_{n}) ‖_{H} \to ‖ (u, v) ‖_{H}$ , as $n \to \infty$ and the proof is finished. □

Hereafter, we denote by B the ball in $R^{N}$ with center 0 and radius $R / 2$ , that is, $B = B_{R / 2} (0)$ , the set $Λ = B_{R} (0)$ and by $I_{0} : H_{0}^{1} (B) \times H_{0}^{1} (B) \to R$ the functional $\begin{array}{rcl} I_{0} (u, v) & = & \frac{1}{2} \int_{B} (| \nabla u |^{2} + max_{B} (V (x), 1) u^{2}) d x \\ + \frac{1}{2} \int_{B} (| \nabla v |^{2} + max_{B} (V (x), 1) v^{2}) d x - \int_{B} K (x) F (u, v) d x . \end{array}$ Moreover, we denote by d the mountain level associated with $I_{0}$ , that is, $\begin{matrix} d = inf_{γ \in Γ} max_{t \in [0, 1]} I_{0} (γ (t)), \end{matrix}$ where $\begin{matrix} Γ = {γ \in C ([0, 1], H_{0}^{1} (B) \times H_{0}^{1} (B)) : γ (0) = 0 and γ (1) = e}, \end{matrix}$ with $e \in H_{0}^{1} (B) \times H_{0}^{1} (B) ∖ {(0, 0)}$ verifying $I_{0} (e) < 0$ .
Remark 3.1.
We observe that $J (u, v) ⩽ I_{0} (u, v)$ for all $u, v \in H_{0}^{1} (B)$ . In particular we have $J (e) ⩽ I_{0} (e) < 0$ . We denote by $m_{J}$ the mountain pass level associated with J, that is, $\begin{matrix} m_{J} = inf_{γ \in Γ} max_{t \in [0, 1]} J (γ (t)), \end{matrix}$ where $\begin{matrix} Γ = {γ \in C ([0, 1], H) : γ (0) = 0 and γ (1) = e} . \end{matrix}$ It is easily seen that $m_{J} ⩽ d$ .

In order to prove the existence of a critical point for J we will use the following version of the mountain pass theorem, which is a consequence of the Ekeland Variational Principle as developed in [19] (see also [7,13]).
Proposition 3.1 (Mountain pass theorem, cf. [15]).

Le X be a Banach space and $Φ \in C^{1} (X, R)$ , $e \in X$ and $r > 0$ such that $‖ e ‖ > r$ and $\begin{matrix} b : = inf_{‖ u ‖ = r} Φ (u) > Φ (0) ⩾ Φ (e) . \end{matrix}$ If Φ satisfies the Palais–Smale condition at level $\begin{matrix} c : = inf_{γ \in Γ} max_{t \in [0, 1]} Φ (γ (t)), \end{matrix}$ where $\begin{matrix} Γ = {γ \in C ([0, 1], X) : γ (0) = 0 and γ (1) = e} . \end{matrix}$ Then c is a critical value of Φ .

Lemma 3.3.
Let be $(u, v) \in H$ a critical point of the functional J at the minimax level $m_{J}$ . Then $(u, v)$ satisfies the estimative $\begin{matrix} {‖ (u, v) ‖}_{H}^{2} ⩽ (\frac{1}{k} - \frac{λ}{k^{2}}) d . \end{matrix}$
Proof.
It is enough to combine (3.1) with definition of d and the fact that $m_{J} ⩽ d$ . □
Lemma 3.4.
If ${(u, v)} \in H$ is a point critical of the functional J, then $u, v ⩾ 0$ .
Proof.
Observe that hence (2.5) and (2.6) it follows that $\begin{array}{rcl} \frac{u^{-} {\hat{F}}_{u} + v^{-} {\hat{F}}_{v}}{\frac{K}{2 k} ({(u^{-})}^{2} + {(v^{-})}^{2})} & = & p η (| K^{1 / 2} (u, v) |) \frac{F (u^{-}, v^{-})}{\frac{K (x)}{2 k} ({(u^{-})}^{2} + {(v^{-})}^{2})} + 2 A (1 - η (| K^{1 / 2} (u, v) |)) \\ + η^{'} (| K^{1 / 2} (u, v) |) K^{1 / 2} ({(u^{-})}^{2} + {(v^{-})}^{2}) {(u^{2} + v^{2})}^{- 1 / 2} \\ \times [\frac{F (u^{-}, v^{-})}{\frac{K (x)}{2 k} ({(u^{-})}^{2} + {(v^{-})}^{2})} - A (x)] . \end{array}$ Since $A (x) \to 0$ uniformly as $a \to 0$ , we have $\begin{matrix} \frac{u^{-} {\hat{F}}_{u} + v^{-} {\hat{F}}_{v}}{\frac{K (x)}{2 k} ({(u^{-})}^{2} + {(v^{-})}^{2})} ⩽ 2 . \end{matrix}$ Consider $Ω = R^{N} ∖ Λ$ . Using that last inequality and from the item 2, Remark 2.1, it follows that $\begin{array}{rcl} \int_{R^{N}} ({| \nabla u^{-} |}^{2} + V (x) ({(u^{-})}^{2} + {(v^{-})}^{2}) + {| \nabla v^{-} |}^{2}) d x \\ ⩽ \int_{Λ} K (x) (u^{-} F_{u} (u, v) + v^{-} F_{v} (u, v)) d x + \int_{Ω} K (x) (u^{-} {\hat{F}}_{u} + v^{-} {\hat{F}}_{v}) d x \\ ⩽ \int_{Ω} (\frac{λ K}{k} ({(u^{-})}^{2} + {(v^{-})}^{2})) d x . \end{array}$ Hence, we have $\begin{array}{rcl} \int_{R^{N}} ({| \nabla u^{-} |}^{2} + V (x) {(u^{-})}^{2} + {| \nabla v^{-} |}^{2} + V (x) {(v^{-})}^{2}) d x \\ ⩽ \int_{R^{N}} \frac{λ V (x)}{k} ({(u^{-})}^{2} + {(v^{-})}^{2}) d x . \end{array}$ Therefore, we conclude $‖ (u^{-}, v^{-}) ‖_{H} = 0$ . □

We have proved up this moment the following result: Proposition 3.2.
There is a critical point $(φ, ϕ) \in H$ with $φ, ϕ$ nonnegative functions associated to the functional $\begin{array}{rcl} J (u, v) & = & \frac{1}{2} {‖ (u, v) ‖}_{H}^{2} - \int_{R^{N}} K (x) G (x, u, v) d x, (u, v) \in H, \end{array}$ at the critical level $\begin{matrix} m_{J} = inf_{γ \in Γ} max_{t \in [0, 1]} J (γ (t)), \end{matrix}$ where $\begin{matrix} Γ = {γ \in C ([0, 1], H) : γ (0) = 0 and γ (1) = e}, \end{matrix}$ with $e \in H_{0}^{1} (B) \times H_{0}^{1} (B) ∖ {(0, 0)}$ verifying, $I_{0} (e) < 0$ .
Proof.
It follows from Proposition 3.1. □

4. Qualitative properties

The next proposition is crucial to our argument. It establishes an important estimate involving the $L^{\infty} (R^{N})$ norm for solutions $(u, v)$ of the system ( AS ). Here, we used the Moser iteration scheme which was adapt to our problem from the classical paper [9].

Proposition 4.1.
Let $h \in L^{q} (R^{N})$ , $2 q > N$ , and $v \in H \subset H^{1} (R^{N})$ be a weak solution of the problem $\begin{matrix} (4.1) & \{\begin{matrix} - Δ u + a (x) u = H_{u} (x, u, v) & in R^{N}, \\ - Δ v + b (x) v = H_{v} (x, u, v) & in R^{N} . \end{matrix} \end{matrix}$ Where $H_{u}, H_{v} : R^{N} \times R \times R \to R$ are a continuous functions verifying $\begin{matrix} | H_{u} (x, t, s) | + | H_{v} (x, t, s) | ⩽ h (x) (| t | + | s |), for all t, s > 0 \end{matrix}$ and a, b are nonnegative continuous functions in $R^{N}$ . Then exists a constant $M = M (q, ‖ h ‖_{L^{q} (R^{N})}) > 0$ such that $\begin{array}{rcl} {‖ (u, v) ‖}_{\infty} & = & max (‖ u ‖_{\infty}, ‖ v ‖_{\infty}) \\ (4.2) & ⩽ & M {‖ max (| u (x) |, | v (x) |) ‖}_{L^{2^{}} (R^{N})} . \end{array}$
Proof.
For each $m \in N$ and $β > 1$ , let us consider the subsets of $R^{N}$ , $\begin{array}{rcl} A_{m} & = & {x \in R^{N} : | v |^{β - 1} ⩽ m} and B_{m} = R^{N} ∖ A_{m}, \end{array}$ and the function $\begin{matrix} v_{m} = \{\begin{matrix} v | v |^{2 (β - 1)} & in A_{m}, \\ m^{2} v & in B_{m} . \end{matrix} \end{matrix}$ Since $v | v |^{2 (β - 1)} = m^{2} v$ on $\partial A_{m}$ , using standard properties of Sobolev spaces we can conclude that $v_{m} \in H^{1} (R^{N})$ . From $\int_{R^{N}} V (x) v_{m}^{2} (x) d x < \infty$ , we have $v_{m} \in H$ , and an easy calculation yields that $\begin{matrix} (4.3) & \nabla v_{m} = \{\begin{matrix} (2 β - 1) | v |^{2 (β - 1)} \nabla v & in A_{m}, \\ m^{2} \nabla v & in B_{m} . \end{matrix} \end{matrix}$ Taking $v_{m}$ as a test function in (4.1) we have, $\begin{matrix} \int_{R^{N}} (\nabla v \nabla v_{m} + b (x) v v_{m}) d x = \int_{R^{N}} H_{v} (x, u, v) v_{m} d x . \end{matrix}$ From (4.3), $\begin{matrix} (4.4) & \int_{R^{N}} \nabla v \nabla v_{m} d x = (2 β - 1) \int_{A_{m}} | v |^{2 (β - 1)} | \nabla v |^{2} d x + m^{2} \int_{B_{m}} | \nabla v |^{2} d x . \end{matrix}$ Considering $\begin{matrix} ω_{m} = \{\begin{matrix} v | v |^{β - 1} & in A_{m}, \\ m v & in B_{m} = R^{N} ∖ A_{m} \end{matrix} \end{matrix}$ by similar argument it follows that $\begin{matrix} (4.5) & \nabla ω_{m} = \{\begin{matrix} β | v |^{β - 1} \nabla v & in A_{m}, \\ m \nabla v & in B_{m} \end{matrix} \end{matrix}$ and $ω_{m} \in H$ . Observe that $\begin{matrix} (4.6) & \int_{R^{N}} | \nabla ω_{m} |^{2} d x = β^{2} \int_{A_{m}} | v |^{2 (β - 1)} | \nabla v |^{2} d x + m^{2} \int_{B_{m}} | \nabla v |^{2} d x . \end{matrix}$ From (4.4), (4.5) and (4.6) $\begin{array}{rcl} \int_{R^{N}} (| \nabla ω_{m} |^{2} + b (x) ω_{m}^{2}) d x - \int_{R^{N}} (\nabla v \nabla v_{m} + b (x) v v_{m}) d x \\ = \int_{A_{m}} (β^{2} - 2 β + 1) | v |^{2 (β - 1)} | \nabla v |^{2} d x . \end{array}$ Using (4.4), we get the inequality $\begin{matrix} (2 β - 1) \int_{A_{m}} | v |^{2 (β - 1)} | \nabla v |^{2} d x ⩽ \int_{R^{N}} (\nabla v \nabla v_{m} + b (x) v v_{m}) d x, \end{matrix}$ which leads to $\begin{matrix} \int_{R^{N}} (| \nabla ω_{m} |^{2} + b (x) ω_{m}^{2} d x) ⩽ [\frac{(β^{2} - 2 β + 1)}{(2 β - 1)} + 1] \int_{R^{N}} (\nabla v \nabla v_{m} + b (x) v v_{m}) d x . \end{matrix}$ Using (4.1), it follows that $\begin{array}{rcl} \int_{R^{N}} (| \nabla ω_{m} |^{2} + b (x) ω_{m}^{2} d x) & ⩽ & \frac{β^{2}}{(2 β - 1)} \int_{R^{N}} H_{v} (x, u, v) v_{m} d x \\ ⩽ & β^{2} \int_{R^{N}} H_{v} (x, u, v) v_{m} d x . \end{array}$ Let S be the best constant which verifies $\begin{matrix} ‖ v ‖_{L^{2^{}} (R^{N})}^{2} ⩽ S \int_{R^{N}} | \nabla v |^{2} d x for all v \in H^{1} (R^{N}) . \end{matrix}$ Using the definition of $v_{m}$ it follows that $| v_{m} | ⩽ | v |^{2 β - 1}$ . Then, $\begin{matrix} | H_{v} (x, u, v) v_{m} | ⩽ 2 h (x) max {(| u (x) |, | v (x) |)}^{2 β} \end{matrix}$ and therefore we have $\begin{matrix} {[\int_{A_{m}} | ω_{m} |^{2^{}}]}^{(N - 2) / N} ⩽ S β^{2} \int_{R^{N}} 2 h (x) max {(| u (x) |, | v (x) |)}^{2 β} d x . \end{matrix}$ If $1 / q_{1} + 1 / q = 1$ , from Holder’s inequality, $\begin{matrix} {[\int_{A_{m}} | ω_{m} |^{2^{}}]}^{(N - 2) / N} ⩽ 2 S β^{2} ‖ h ‖_{L^{q} (R^{N})} {[\int_{R^{N}} {| max (| u (x) |, | v (x) |) |}^{2 β q_{1}} d x]}^{1 / q_{1}} . \end{matrix}$ Since $| ω_{m} | ⩽ | v |^{β} in R^{N}$ and $| ω_{m} | = | v |^{β} in A_{m}$ , it follows that $\begin{matrix} {[\int_{A_{m}} {(| v |^{β})}^{2^{}}]}^{(N - 2) / N} ⩽ 2 S β^{2} ‖ h ‖_{L^{q} (R^{N})} {[\int_{R^{N}} {| max (| u (x) |, | v (x) |) |}^{2 β q_{1}} d x]}^{1 / q_{1}} . \end{matrix}$ Taking the limit as $m \to \infty$ and using monotone convergence theorem, we obtain $\begin{matrix} ‖ v ‖_{2^{} β}^{2 β} ⩽ 2 S β^{2} ‖ h ‖_{L^{q} (R^{N})} {‖ max (| u (x) |, | v (x) |) ‖}_{2 β q_{1}}^{2 β} \end{matrix}$ and $\begin{array}{rcl} (4.7) & ‖ v ‖_{2^{} β} & ⩽ & β^{1 / β} {(2 S ‖ h ‖_{L^{q} (R^{N})})}^{1 / (2 β)} {‖ max (| u (x) |, | v (x) |) ‖}_{2 β q_{1}} . \end{array}$ Since $N / (N - 2) > q_{1}$ , set $σ = N / (q_{1} (N - 2))$ . When $β = σ in$ (4.7), we have: $2 q_{1} β = 2^{}$ and $\begin{array}{rcl} (4.8) & ‖ v ‖_{2^{} σ} & ⩽ & σ^{1 / σ} {(2 S ‖ h ‖_{L^{q} (R^{N})})}^{1 / (2 σ)} {‖ max (| u (x) |, | v (x) |) ‖}_{2^{}} . \end{array}$ When $β = σ^{2}$ in (4.7) we get $2 q_{1} β = 2^{} σ$ and $\begin{array}{rcl} (4.9) & ‖ v ‖_{2^{} σ^{2}} & ⩽ & σ^{2 / σ^{2}} {(2 S ‖ h ‖_{L^{q} (R^{N})})}^{1 / (2 σ^{2})} {‖ max (| u (x) |, | v (x) |) ‖}_{2^{} σ} . \end{array}$ The inequalities (4.8) and (4.9) imply that $\begin{matrix} (4.10) & ‖ v ‖_{2^{} σ^{2}} ⩽ σ^{\frac{1}{σ} + \frac{2}{σ^{2}}} {(2 S ‖ h ‖_{L^{q} (R^{N})})}^{\frac{1}{2} (\frac{1}{σ} + \frac{1}{σ^{2}})} {‖ max (| u (x) |, | v (x) |) ‖}_{2^{}} . \end{matrix}$ An iteration argument, replacing β by $σ^{j}$ in (4.7), lead us to $\begin{matrix} (4.11) & ‖ v ‖_{2^{} σ^{j}} ⩽ σ^{\sum_{l = 1}^{j} \frac{l}{σ^{l}}} {(2 S ‖ h ‖_{L^{q} (R^{N})})}^{\frac{1}{2} (\sum_{l = 1}^{j} \frac{1}{σ^{l}})} {‖ max (| u (x) |, | v (x) |) ‖}_{2^{}} . \end{matrix}$ Once that $\begin{array}{rcl} \sum_{j = 1}^{\infty} \frac{j}{σ^{j}} = \frac{σ}{{(σ - 1)}^{2}}, \\ \frac{1}{2} \sum_{j = 1}^{\infty} \frac{1}{σ^{j}} = \frac{1}{2} \frac{1}{(σ - 1)}, \end{array}$ it follows from (4.11) $\begin{matrix} ‖ v ‖_{i} ⩽ σ^{1 / (σ - 1)} {(2 S ‖ h ‖_{L^{q} (R^{N})})}^{1 / (2 (σ - 1))} {‖ max (| u (x) |, | v (x) |) ‖}_{2^{}}, \end{matrix}$ for all $i ⩾ 2^{}$ . Since $\begin{matrix} ‖ v ‖_{\infty} = lim_{i \to \infty} ‖ v ‖_{i}, \end{matrix}$ we get $\begin{matrix} ‖ v ‖_{\infty} ⩽ M {‖ max (| u (x) |, | v (x) |) ‖}_{2^{}}, \end{matrix}$ where $\begin{matrix} M = max (1, σ^{σ / {(σ - 1)}^{2}} {(2 S ‖ h ‖_{q})}^{1 / (2 (σ - 1))}) with σ = \frac{N (q - 1)}{q (N - 2)} . \end{matrix}$ Using an argument similar to the used above, we can also prove that $\begin{matrix} ‖ u ‖_{\infty} ⩽ M {‖ max (| u (x) |, | v (x) |) ‖}_{2^{}} \end{matrix}$ which complete the proof of Proposition 4.1. □
Lemma 4.1.
If $(u, v) \in H$ is a critical point of the functional J, then* $u, v > 0$ .
Proof.
If $(u, v) \in H$ is a critical point of J, from Lemma 3.4 we have $u, v ⩾ 0$ . By Proposition 4.1 we obtain $u, v \in L^{\infty} (R^{N})$ , and therefore the result follows with the help of the maximum principle. □
Lemma 4.2.
For any $R > 1$ , any bound state solution $(u, v)$ with $u, v > 0$ of ( AS ) satisfies $\begin{array}{rcl} {‖ (u, v) ‖}_{\infty} & ⩽ & \tilde{M} {((\frac{1}{k} - \frac{λ}{k^{2}}) S d)}^{1 / 2} . \end{array}$
Proof.
If $(u, v)$ is a bound state solution of ( AS ) with $u, v > 0$ , then it satisfies the following system $\begin{matrix} \{\begin{matrix} - Δ u + b (x) u = H_{u} (x, u, v) & in R^{N}, \\ - Δ v + b (x) v = H_{v} (x, u, v) & in R^{N}, \\ u, & v \in H^{1} (R^{N}), \end{matrix} \end{matrix}$ where $H_{u}, H_{v} : R^{N} \times R \times R \to R$ and $b : R^{N} \to R$ are given by $\begin{array}{rcl} H_{u} (x, t, s) = χ_{Λ} (x) K (x) F_{u} (t, s) + (1 - χ_{Λ} (x)) η (| K {(x)}^{1 / 2} (u, v) |) K (x) F_{u} (u, v) \\ + (1 - χ_{Λ} (x)) η^{'} (| K {(x)}^{1 / 2} (u, v) |) \frac{u}{{(u^{2} + v^{2})}^{1 / 2}} K {(x)}^{1 / 2} F (u, v), \\ H_{v} (x, t, s) = χ_{Λ} (x) K (x) F_{v} (t, s) + (1 - χ_{Λ} (x)) η (| K {(x)}^{1 / 2} (u, v) |) K {(x)}^{1 / 2} F_{v} (u, v) \\ + (1 - χ_{Λ} (x)) η^{'} (| K {(x)}^{1 / 2} (u, v) |) \frac{v}{{(u^{2} + v^{2})}^{1 / 2}} K (x) F (u, v), \\ b (x) = V (x) - (1 - χ_{Λ} (x)) \frac{1}{k} K (x) A (x) (1 - η (| K {(x)}^{1 / 2} (u, v) |)) \\ + (1 - χ_{Λ} (x)) η^{'} (| K {(x)}^{1 / 2} (u, v) |) \frac{K {(x)}^{1 / 2}}{{(u^{2} + v^{2})}^{1 / 2}} [A (x) K (x) \frac{1}{2 k} (u^{2} + v^{2})], \end{array}$ where $χ_{Λ}$ is the characteristic function of the set Λ and η was defined in (2.2). It is easy to see that $H_{u}$ and $H_{v}$ are continuous functions.

Taking a sufficiently small, we can conclude that $b (x)$ is a nonnegative function, $\begin{matrix} \int_{R^{N}} b (x) (u^{2} + v^{2}) d x < + \infty . \end{matrix}$ Besides that, $V (x)$ is a bounded function and using the Remark 2.1, we have the following inequalities $\begin{array}{rcl} | H_{u} (x, t, s) | & = & | χ_{Λ} (x) K (x) F_{u} (t, s) | + | (1 - χ_{Λ} (x)) η (| {(K (x))}^{1 / 2} (t, s) |) K (x) F_{u} (t, s) | \\ + | (1 - χ_{Λ} (x)) η^{'} (| {(K (x))}^{1 / 2} (t, s) |) t {(t^{2} + s^{2})}^{- 1 / 2} {(K (x))}^{1 / 2} F (t, s) | \\ ⩽ & C_{1} (| t |^{(p - 2)} + | s |^{(p - 2)}) (| t | + | s |) + \frac{C_{2}}{5 a^{2}} (| t |^{p} + | s |^{p}) t \\ ⩽ & C_{3} (| t |^{(p - 2)} + | s |^{(p - 2)}) (| t | + | s |) \end{array}$ and $\begin{array}{rcl} | H_{v} (x, t, s) | & = & | χ_{Λ} (x) K (x) F_{v} (t, s) | + | (1 - χ_{Λ} (x)) η (| {(K (x))}^{1 / 2} (t, s) |) K (x) F_{v} (t, s) | \\ + | (1 - χ_{Λ} (x)) η^{'} (| {(K (x))}^{1 / 2} (t, s) |) s {(t^{2} + s^{2})}^{- 1 / 2} {(K (x))}^{1 / 2} F (t, s) | \\ ⩽ & C_{1} (| t |^{(p - 2)} + | s |^{(p - 2)}) (| t | + | s |) + \frac{C_{2}}{5 a^{2}} (| t |^{p} + | s |^{p}) s \\ ⩽ & C_{3} (| t |^{(p - 2)} + | s |^{(p - 2)}) (| t | + | s |), \end{array}$ where $C_{1}$ , $C_{2}$ are real positive constants such that $C_{1} = 2 c_{0} {sup}_{Λ} K (x)$ , $C_{2} = 30 c_{0} / (p μ)$ and $C_{3} = max (C_{1}, C_{2})$ . Therefore, $\begin{matrix} | H_{u} (x, u, v) | + | H_{v} (x, u, v) | ⩽ | h (x) | (| u | + | v |) in R^{N} . \end{matrix}$ With h given by $\begin{matrix} h (x) = C {(max (u (x), v (x)))}^{(p - 2)}, \end{matrix}$ where C is a positive constant such that $C = 2 C_{3}$ . A direct computation shows that $h \in L^{q} (R^{N})$ for $q = \frac{2^{}}{p - 2}$ . From the Sobolev inequality and the Lemma 3.3, we get $\begin{array}{rcl} ‖ h ‖_{q} & ⩽ & C {‖ (u, v) ‖}_{2^{}}^{(p - 2)} ⩽ C {(S^{1 / 2} {‖ (u, v) ‖}_{H})}^{(p - 2)} ⩽ C {(S (\frac{1}{k} - \frac{λ}{k^{2}}) d)}^{(p - 2) / 2} . \end{array}$ Now, from the proof of Proposition 4.1, we get $\begin{array}{rcl} M & = & max (1, σ^{σ / {(σ - 1)}^{2}} {(2 S ‖ h ‖_{q})}^{1 / (2 (σ - 1))}) \\ ⩽ & max (1, σ^{σ / {(σ - 1)}^{2}} {(2 S C {(S (\frac{1}{k} - \frac{λ}{k^{2}}) d)}^{(p - 2) / 2})}^{1 / (2 (σ - 1))}) : = \tilde{M} . \end{array}$ Observe that $\tilde{M}$ does not depend on R, u or v. □
Lemma 4.3.
Let $(u, v) \in H$ a bound state solution for the system ( AS ) with u, v positive functions and the potential V is nonnegative bounded continuous functions satisfying $(V_{0})$ – $(V_{1})$ . Then u and v satisfy $\begin{matrix} | max (u (x), v (x)) | ⩽ \frac{R^{N - 2} ‖ (u, v) ‖_{\infty}}{| x |^{N - 2}} ⩽ \frac{R^{N - 2} \tilde{M} {(S (\frac{1}{k} - \frac{λ}{k^{2}}) d)}^{1 / 2}}{| x |^{N - 2}} \end{matrix}$ for all $x \in R^{N} ∖ B_{R} (0)$ .
Proof.
Consider the harmonic function $ψ : (R^{N} ∖ B_{R} (0)) ⟶ R$ such that $\begin{matrix} ψ (x) = \frac{R^{N - 2} ‖ (u, v) ‖_{\infty}}{| x |^{N - 2}} . \end{matrix}$ Observe that the function ψ satisfies $\begin{array}{rcl} - Δ ψ & = & 0 . \end{array}$ Therefore $\begin{matrix} - Δ ψ + (1 - \frac{1}{k}) V (x) ψ ⩾ 0 . \end{matrix}$ Suppose $p > 1$ , we have that the function $ζ : R^{N} \to R$ , $ζ (x) = | x |^{p}$ is convex, thus $\begin{matrix} (| x |^{p - 2} x - | y |^{p - 2} y) (x - y) ⩾ 0 for all x, y \in R^{N} . \end{matrix}$ Now, take as a test function $\begin{matrix} ϕ = \{\begin{matrix} {(u - ψ)}^{+} & if x \in R^{N} ∖ B_{R} (0), \\ 0 & if x \in B_{R} (0), \end{matrix} and φ = \{\begin{matrix} {(v - ψ)}^{+} & if x \in R^{N} ∖ B_{R} (0), \\ 0 & if x \in B_{R} (0) . \end{matrix} \end{matrix}$ Observe that $\begin{matrix} u ⩽ ψ in \partial B_{R} and v ⩽ ψ in \partial B_{R} (0) \end{matrix}$ then we can conclude that $ϕ, φ \in H^{1} (R^{N})$ . Besides that $\begin{matrix} J^{'} (u, v) (ϕ, φ) - \int_{R^{N} ∖ B_{R} (0)} \frac{1}{k} (V (x) u ϕ + V (x) v φ) ⩽ 0 . \end{matrix}$ Consider $Ω : = {x \in R^{N} ∖ B_{R} (0) : min {u, v} > ψ}$ . Hence, combining these estimates $\begin{array}{rcl} 0 & ⩾ & \int_{Ω} (\nabla u - \nabla ψ) \nabla ϕ + (1 - \frac{1}{k}) V (x) (u - ψ) ϕ d x \\ + \int_{Ω} (\nabla v - \nabla ψ) \nabla φ + (1 - \frac{1}{k}) V (x) (v - ψ) φ d x \\ ⩾ & \int_{Ω} (1 - \frac{1}{k}) [V (x) (u - ψ) (u - ψ) + V (x) (v - ψ) (v - ψ)] d x . \end{array}$

Therefore, the set $Ω = {x \in R^{N} ∖ B_{R} (0) : min {u, v} > ψ}$ is empty. Then the proof is complete. □

5. Proof of Theorem 1.1 completed

Proof.

From Lemmas 3.1 and 3.2, problem ( AS ) has a bound state solution $(u, v) \in H^{1} (R^{N}) \times H^{1} (R^{N})$ with $u, v$ positive functions. Therefore, it is enough to show that $(u, v)$ satisfies the inequality $| K {(x)}^{1 / 2} (u, v) | ⩽ a$ . Using the hyphotesis $(V_{2})$ , $\begin{matrix} K (x) (| u |^{2} + | v |^{2}) ⩽ 2 K (x) \frac{R^{2 (N - 2)} {\tilde{M}}^{2} (S (\frac{1}{k} - \frac{λ}{k^{2}}) d)}{| x |^{2 (N - 2)}} ⩽ γ {\tilde{M}}^{2} 2 S (\frac{1}{k} - \frac{λ}{k^{2}}) d \end{matrix}$ for all $| x | ⩾ R$ . Consider $γ^{*} = a^{2} / ({\tilde{M}}^{2} 2 S (\frac{1}{k} - \frac{λ}{k^{2}}) d)$ . Observe that the constant $\tilde{M}$ not depend on R, v or u, see Lemma 4.2. So, for all γ with $0 < γ ⩽ γ^{*}$ , we have that ( AS )-auxiliary system-solution $(u, v)$ is also ( S )-system principle, because $η (| K {(x)}^{1 / 2} (u, v) |) \equiv 1$ for all $| x | ⩾ R$ . Hence, we complete the proof. □

Footnotes

Acknowledgements

The authors would like to thank the anonymous referee for suggestions and valuable comments, which has been significantly improved the manuscript.

Research partially supported by the National Institute of Science and Technology of Mathematics INCT-Mat, CAPES and CNPq.

References

[1]

C.O.

Alves , Local mountain pass for a class of elliptic system, J. Math. Anal. Appl. 335 (2007), 135–150.

[2]

C.O.

Alves ,

J.M.

do Ó and

M.A.S.

Souto , Local mountain pass for a class of elliptic problems involving critical growth, Nonlinear Anal. 46 (2001), 495–510.

[3]

C.O.

Alves ,

G.M.

Figueiredo and

M.F.

Furtado , Multiplicity of solutions for elliptic systems via local mountain pass method, Commun. Pure Appl. Anal. 8 (2009), 1745–1758.

[4]

C.O.

Alves and

M.A.S.

Souto , Existence of solutions f or a class of elliptic equations in

R^{N}

with vanishing potentials, J. Differential Equations 252 (2012), 5555–5568.

[5]

Ambrosetti ,

Felli and

Malchiodi , Ground states of nonlinear Schrödinger equations with potentials vanishing at infinity, J. Eur. Math. Soc. 7 (2005), 117–144.

[6]

Ambrosetti and

Z.-Q.

Wang , Nonlinear Schrödinger equations with vanishing and decaying potentials, Differential Integral Equations 18 (2005), 1321–1332.

[7]

J.P.

Aubin and

Ekeland , Applied Nonlinear Analysis, Pure and Applied Mathematics, John Wiley & Sons, Inc., New York, 1984. A Wiley-Interscience Publication.

[8]

Berestycki and

P.L.

Lions , Nonlinear scalar field equations I. Existence of a ground state, Arch. Rational Mech. Anal. 82 (1983), 313–345.

[9]

Brezis and

Kato , Remarks on the Schröndinger operator with regular complex potentials, J. Math Pures Appl. 58 (1979), 137–151.

10.

[10]

Costa , On a class of elliptic systems in

R^{N}

, Electron. J. Differential Equations 7 (1994), 1–14.

11.

[11]

del Pino and

Felmer , Local mountain pass for semilinear elliptic problems in unbounded domains, Calc. Var. Partial Differential Equations 4 (1996), 121–137.

12.

[12]

Ding and

W.M.

Ni , On the existence of positive entire solutions of a semilinear elliptic equation, Arch. Rational Mech. Anal. 91 (1986), 283–308.

13.

[13]

Kavian , Introduction à la théorie des points critiques et applications aux problèmes elliptiques, Mathématiques & Applications, Vol. 13, Springer-Verlag, Paris, 1993.

14.

[14]

P.-L.

Lions , The concentration-compactness principle in the calculus of variations. The limit case. II, Rev. Mat. Iberoamericana 1 (1985), 45–121.

15.

[15]

P.H.

Rabinowitz , Minimax Methods in Critical Point Theory with Applications to Differential Equations, CBMS, Vol. 65, AMS, 1986.

16.

[16]

P.H.

Rabinowitz , On a class of nonlinear Schrödinger equations, Z. Angew. Math. Phys. 43 (1992), 270–291.

17.

[17]

Sirakov , Existence and multiplicity of solutions of semilinear elliptic equations in

R^{N}

, Cal. Var. Partial Differential Equations 11 (2000), 119–142.

18.

[18]

Su ,

Z.-Q.

Wang and

Willem , Nonlinear Schrödinger equations with unbounded and decaying radial potentials, Commun. Contemp. Math. 9(4) (2007), 571–583.

19.

[19]

Willem , Minimax Theorems, Progress in Nonlinear Differential Equations and Their Applications, Vol. 24, Birkhäuser Boston, Inc., Boston, MA, 1996.