Intermediate symplectic characters and shifted plane partitions of shifted double staircase shape

We use intermediate symplectic characters to give a proof and variations of Hopkins' conjecture, now proved by Hopkins and Lai, on the number of shifted plane partitions of shifted double staircase shape with bounded entries. In fact, we prove some character identities involving intermediate symplectic characters, and find generating functions for such shifted plane partitions. The key ingredients of the proof are a bialternant formula for intermediate symplectic characters, which interpolates between those for Schur functions and symplectic characters, and the Ishikawa-Wakayama minor-summation formula.


Introduction
This paper is motivated by a conjecture of Hopkins, now proved by Hopkins-Lai [13], on the number of shifted plane partitions of shifted double staircase shape with bounded entries (see Theorem 1.1 below). The goal of this paper is to give a proof and variations (including q-analogues) by using intermediate symplectic characters.
Given a strict partition µ = (µ 1 , . . . , µ l ), the shifted diagram S(µ) of µ is defined to be the array of unit squares with µ i squares in the ith row from top to bottom such that each row is indented by one square to the right with respect to the previous row. A shifted plane partition of shape µ is a filling of the cells of S(µ) with nonnegative integers such that the entries in each row and column are weakly decreasing. For a nonnegative integer m, we denote by A m (S(µ)) the set of all shifted plane partitions of shape µ with entries bounded by m. We write δ r = (r, r − 1, . . . , 2, 1). Hopkins and Lai [13] prove the following product formula by counting lozenge tilings of a certain region in the triangular lattice. For 0 ≤ k ≤ n, the number of shifted plane partitions of shifted double staircase shape δ n + δ k = (n + k, n + k − 2, . . . , n − k + 2, n − k, n − k − 1, . . . , 2, 1) with entries bounded by m is given by #A m (S(δ n + δ k )) = 1≤i≤j≤n m + i + j − 1 i + j − 1 1≤i≤j≤k m + i + j i + j . (1.1) In this paper, we shall prove and generalize this formula by establishing identities involving intermediate symplectic characters. Our algebraic approach is inspired by the proofs of the k = 0 case due to Macdonald [21] and the k = n case due to Proctor [25].
If k = 0, then shifted plane partitions of shifted staircase shape δ n with entries bounded by m are in one-to-one correspondence with symmetric plane partitions contained in the n×n×m box. Then we have two different q-analogues of (1.1); one is the MacMahon conjecture [22, p. 153], proved by Andrews [1,2] and Macdonald [21]: , (1.2) and the other is the Bender-Knuth conjecture [6,Eq. (8)], proved by Andrews [3], Gordon [12] and Macdonald [21]: Here we use the notation σ = 1 2 n i=1 σ i,i + 1≤i<j≤n σ i,j , |σ| = 1≤i≤j σ i,j and [r] = (1 − q r )/(1 − q). Macdonald's proof [21,I.5 Examples 16,17 and 19] of these identities proceeds as follows. By transforming shifted plane partitions into semistandard tableaux, the left hand sides of (1.2) and (1.3) are expressed in terms of the q-specializations of Schur functions s λ (x 1 , . . . , x n ): where the sums are taken over all partitions λ whose Young diagrams are contained in the m × n rectangle, i.e., the diagram of (m n ). In this setting, the key role is played by the following character identity: where o B (m n ) (x 1 , . . . , x n ) is an odd orthogonal character, which is an irreducible character of O 2n+1 , the double-cover of the odd orthogonal group O 2n+1 . Then we can obtain (1.2) and (1.3) by using the q-analogues of the Weyl dimension formula.
In a similar vein, Proctor [25] derived the case k = n of (1.1) from the character identity λ⊂(m n ) sp λ (x 1 , . . . , x n ) = s (m n ) (x 1 , x −1 1 , . . . , x n , x −1 n , 1), (1.5) where sp λ (x 1 , . . . , x n ) is a symplectic character, which is an irreducible character of the symplectic group Sp 2n . Now we explain our proof of Theorem 1.1. The main actor is a family of intermediate symplectic characters sp (k,n−k) λ (x 1 , . . . , x k |x k+1 , . . . , x n ), introduced by Proctor [26], which are defined as the multivariate generating functions of (k, n − k)-symplectic tableaux (see Definition 2.1). In the extreme cases, they reduce to the Schur functions s λ (x) = sp (0,n) λ (x) and the symplectic characters sp λ (x) = sp (n,0) λ (x). As a special case of our main theorem (Theorem 3.1), we obtain the following character identity. Note that Equation (1.6) reduces to (1.4) and (1.5) when k = 0 and n respectively (see also Corollary 3.2). The proof of our main theorem (Theorem 3.1) is a generalization of proofs of (1.4) and (1.5) provided in [23], and based on the Ishikawa-Wakayama minor summation formula [14]. An additional key ingredient of the proof is a bialternant formula for intermediate symplectic characters (Theorem 2.8), which is another main result of this paper. Theorem 1.2 enables us to find q-analogues of (1.1). Given a shifted plane partition σ ∈ A m (δ n + δ k ), we define its weights v(σ) and w(σ) by putting v(σ) = k − is the lth trace of σ.
Since shifted plane partitions of shape δ n + δ k are in bijection with (k, n − k)-symplectic tableaux (see Lemma 5.2), we obtain the following q-analogues of (1.1) by specializing x i = q i or x i = q i−1/2 in (1.6). Corollary 1.3. For 0 ≤ k ≤ n, the generating functions of shifted plane partitions of shifted double staircase shape δ n + δ k with entries bounded by m are given by (1.11)

Jacobi-Trudi and bialternant formulas
In this section, we recall a definition of intermediate symplectic characters and prove Jacobi-Trudi and bialternant formulas for them.

Intermediate symplectic characters
A partition is a weakly decreasing sequence λ = (λ 1 , λ 2 , . . . ) of nonnegative integers with only finitely many nonzero entries. A partition λ is usually represented by its Young diagram D(λ), which is the left-justified array of unit squares with λ i squares in the ith row. The length of a partition λ, denoted by l(λ), is the number of nonzero entries of λ.
Proctor [26] introduced the notion of intermediate symplectic tableaux to describe weight bases for indecomposable representations of the intermediate symplectic groups. For 0 ≤ k ≤ n, the intermediate symplectic group Sp 2k,n−k is defined to be the subgroup of the general linear group GL n+k which preserves a skew-symmetric bilinear form of rank 2k. Then we have Sp 0,n = GL n and Sp 2n,0 = Sp 2n . Definition 2.1. Let 0 ≤ k ≤ n and λ a partition of length ≤ n. A (k, n − k)-symplectic tableau of shape λ is a filling of the cells of the Young diagram D(λ) with entries from Γ k,n−k = {1 < 1 < 2 < 2 < · · · < k < k < k + 1 < k + 2 < · · · < n} satisfying the following three conditions: (i) the entries in each row are weakly increasing; (ii) the entries in each column are strictly increasing; (iii) the entries of the ith row are greater than or equal to i.
We denote by SpTab (k,n−k) (λ) the set of (k, n − k)-symplectic tableaux of shape λ. Given a (k, n − k)-symplectic tableau T , we define , where x = (x 1 , . . . , x n ) are indeterminates and m T (γ) denotes the multiplicity of γ ∈ Γ k,n−k in T . Then the (k, n − k)-symplectic character corresponding to λ is defined by (2.1) For example, is a (2, 2)-symplectic tableau of shape (4, 3, 1, 1) and . The (0, n)-symplectic tableaux are the same as ordinary semistandard tableaux, while the (n, 0)-symplectic tableaux are the same as King's symplectic tableaux [15]. Hence the Schur function s λ (x) and the symplectic character sp λ (x) can be defined as the extremal cases (k = 0 and k = n) of the intermediate symplectic characters: When k = n − 1, the (n − 1, 1)-symplectic characters are called odd symplectic characters. The (k, n − k)-symplectic characters are collectively referred to as the intermediate symplectic characters.
In general, the (k, n − k)-symplectic character sp (k,n−k) λ can be expressed as where µ runs over all partitions of length ≤ k such that µ ⊂ λ, and s λ/µ is the skew Schur function. If λ = (r n ) is a rectangular partition, then we have the following factorization formula.
Proposition 2.2. For a nonnegative integer r, we have Proof. If T is a (k, n − k)-symplectic tableau of shape (r n ), then the first k rows of T form a (k, 0)-symplectic tableau of shape (r k ) and the ith row contains only the letter i for k + 1 ≤ i ≤ n.
Note that the denominators factor as follows: and And the following determinant formulas are referred to as the Jacobi-Trudi identities:

Jacobi-Trudi identities
The following Jacobi-Trudi identity can be obtained by using a lattice path interpretation of (k, n − k)-symplectic tableaux and the Lindström-Gessel-Viennot lemma (see [19] and [10,11]). Proposition 2.3. For a partition λ of length ≤ n, we have . . , x n ) stands for the rth complete symmetric polynomial in Proof. Let G = (V, E) be the directed graph with vertex set and edges directed from u to v whenever v − u = (1, 0) or (0, 1). Given u, v ∈ V , we denote by L(u; v) the set of all lattice paths from u to v, i.e., all sequences (w 0 , w 1 , . . . , w r ) of vertices of G such that w 0 = u, w r = v and (w i−1 , w i ) ∈ E for 1 ≤ i ≤ r. A family (P 1 , . . . , P n ) of lattice paths P i is called non-intersecting if no two of them have a vertex in common. We introduce an edge weight wt by putting and wt((i, j), (i, j + 1)) = 1. Then the weights of a lattice path P = (w 0 , w 1 , . . . , w r ) and a family of lattice paths (P 1 , . . . , P n ) are defined by wt(P ) = r i=1 wt(w i−1 , w i ) and wt(P 1 , . . . , P n ) = n i=1 wt(P i ) respectively. For a partition λ of length ≤ n, we put and denote by L 0 (u 1 , . . . , u n ; v 1 , . . . , v n ) the set of non-intersecting lattice paths (P 1 , . . . , P n ) such that P i ∈ L(u i ; v i ) for 1 ≤ i ≤ n. Then there is a weight-preserving bijection between SpTab (k,n−k) (λ) and L 0 (u 1 , . . . , u l ; v 1 , . . . , v l ). See Figure 1 for an example in the case where n = 4, k = 2 and λ = (4, 3, 1, 1). Since the generating function of L(u j ; v i ) is given by we can complete the proof by applying the Lindström-Gessel-Viennot lemma.
By performing column operations, we can deduce the following Jacobi-Trudi-type expression from Proposition 2.3. Proposition 2.4. For a partition λ of length ≤ n, let H (k,n−k) λ be the n × n matrix with (i, j) entry given by Then we have (2.11) When k = 0 (resp. k = n), this proposition reduces to the Jacobi-Trudi identity (2.8) for Schur functions (resp. (2.9) for symplectic characters).
Proof. Recall that the generating function of complete symmetric functions h r (z 1 , . . . , z m ) is given by It follows from this generating function that for k + 1 ≤ j ≤ n. We apply the following two types of column operations to the matrix on the right hand side of (2.10): (a) add the jth column multiplied by x j + x −1 j to the (j + 1)st column for 1 ≤ j ≤ k − 1; (b) add the jth column multiplied by x j to the (j + 1)st column for k + 1 ≤ j ≤ n − 1.
Then, by using the above relations, we can show is the n × n matrix whose (i, j) entry is given by Now, by subtracting the jth column multiplied by 2p+j−1 p from the (2p + j)th column for j = 1, 2, . . . , k and p = 1, 2, . . . , (k − j)/2 , we conclude that det K By the same argument as in the proof of Proposition 2.4, we can show the following: . (2.12) Following [16, Definition 2.1.1], we define the universal symplectic Schur function s C λ (X) as a symmetric function in X = {x 1 , x 2 , . . . } by where h r (X) is the rth complete symmetric function in X. Comparing this with (2.12), we have Corollary 2.6. If l(λ) ≤ k + 1, then we have
Proposition 2.7. For a partition λ of length ≤ n, let A (k,n−k) λ = (a i,j ) be the n × n matrix with (i, j) entry given by Then we have When k = 0 (resp. k = n), this theorem reduces to the bialternant formula (2.4) (resp. (2.5)). If k = n − 1, then we can transform this bialternant formula (2.14) into the bialternant formula obtained in [24, Theorem 1.1].
Proof. Let M (k,n−k) = (m i,j ) be the n × n matrices with (i, j) entry given by where e r (z 1 , . . . , z m ) is the rth elementary symmetric polynomial in z 1 , . . . , z m , and we use the abbreviation e r (z ±1 1 , . . . , z ±1 m ) = e r (z 1 , z −1 1 , . . . , z m , z −1 m ). Then we claim that We prove (2.16) by computing the entries of the product H Equating the coefficient of u r+n , we obtain is a upper-triangular matrix with diagonal entries 1, the special case λ = ∅ of (2.16) gives (2. 19) In particular, we have Hence, by using (2.6) and (2.7), we see that det A is given by (2.15). Also it follows from (2.10), (2.16) and (2.19 For our application it is convenient to convert the bialternant formula in Proposition 2.7 into the following form. Theorem 2.8. For a partition λ of length ≤ n, let A λ = (a i,j ) be the n × n matrix with (i, j) entry given by Then we have Equating the coefficients of u r , we obtain Hence we have .

By using this relation, we can show that the matrix
by adding the last (n − k) columns multiplied by appropriate factors to the first k columns.

Character identities
In this section we state our main theorem, which gives formulas for certain summations of intermediate symplectic characters, and use the Ishikawa-Wakayama minor summation formula to express these summations in terms of Pfaffians. The proof of the main theorem is completed in the next section.
Now we can state the main theorem of this paper. (1) We have For a positive integer m, we have Proof. The proof is accomplished by combining Theorem 3.1 (1), (3) and (4), and the following factorization formulas for rectangular Schur functions: and, for a positive integer m, These identities can be proved in a way similar to that of [4].
The strategy of the proof of Theorem 3.1 is the same as in [23]. Namely, (a) First we use the bialternant formula in Theorem 2.8 and apply the Ishikawa-Wakayama minor-summation formula (Proposition 3.4) to express the summations in Theorem 3.1 in terms of Pfaffians.
(b) Next we transform the resulting Pfaffians into the products of two determinants (see Section 4).

Reduction to the even case
Now we start the proof of Theorem 3.1. The following lemma enables us to reduce the proof of the odd case to that of the even case.
where the symbol f | x 1 =0 stands for the substitution x 1 = 0 in f .
By using this lemma, we deduce the odd case of Theorem 3.1 from the even case as follows. Assume that Equation (3.5) holds for a fixed n and aim to prove the same equation with n replaced by n − 1. Multiplying both sides of (3.5) by x a+m Then by specializing x 1 = 0 and using Lemma 3.3, we obtain µ∈P((m n−1 )) sp (k−1,n−k) µ+(a n−1 ) (x 2 , . . . , x k |x k+1 , . . . , x n ) which is Equation (3.5) with n replaced by n − 1. The other equations (3.6), (3.7) and (3.8) are treated in the same manner. Therefore it suffices to give a proof of Theorem 3.1 in the case n is even.

Minor-summation formula
In the remaining of this section, we assume that n is even.

(3.12)
Then we can express the summations in Theorem 3.1 in terms of the Pfaffian of the skew-symmetric matrix Q n,k (x; a, b). Proposition 3.7. Let n be an even integer and m > 0. We write x r = (x r 1 , . . . , x r n ) and x r [k] = (x r 1 , . . . , x r k ).
Proof. As the proofs are similar, we give a sketch of the proof of (1). By using Theorem 2.
A direct computation gives us By replacing x with x −1 or/and y with y −1 , we obtain .
Using these relations, we can explicitly compute the entries of XB t X. By a straightforward computation, we see that the (i, j) entry of XB t X is equal to Hence, by using the multilinearilty of Pfaffians, we obtain

Pfaffian identity
In this section, we complete the proof of Theorem 3.1 by establishing a Pfaffian identity.

Pfaffian identity
By Proposition 3.7, we need to evaluate the Pfaffian Pf Q n,k (x; a, b) in order to prove Theorem 3.1. For indeterminates y = (y 1 , . . . , y k ) and b = (b 1 , . . . , b k ), let U k,n (y; b) be the k × k matrix with the ith row Then the Pfaffian Pf Q n,k (x; a, b) is evaluated as follows.
In the extreme cases k = 0 and k = n, the skew-symmetric matrix Q n,k (x; a, b) becomes respectively. In these cases, the assertions of Combining these relations together with Proposition 3.7 and Theorem 4.2, we arrive at the desired identities.

Proof of Theorem 4.2
It remains to prove Theorem 4.2.
Since both sides of (4.1) have degree at most one in each of the variables a 1 , . . . , a n , b 1 , . . . , b k , it is enough to show that the coefficients of a I b J = i∈I a i j∈J b j are the same on both sides for any subsets I ⊂ [n] and J ⊂ [k]. We denote by L(I, J) and R(I, J) the corresponding coefficients on the left and right hand sides respectively.
First we compute the coefficient R(I, J). We put Then we can see that the coefficient of a I in det W n (x; a) is equal to where Σ(I) = i∈I i, and that the coefficient where Σ(J) = j∈J j. We put n,k = {(i, j) : k + 1 ≤ i < j ≤ n}.

(4.5)
Then the denominator on the right hand side of (4.1) is written as Since we have it follows from (4.3) and (4.4) that the coefficient of a I b J in the right hand side of (4.1) is given by n,k ∩D + n (I) Next we compute the coefficient L(I, J) on the left hand side of (4.1). By the multilinearity of Pfaffians, we see that L(I, J) is equal to the Pfaffian of the skew-symmetric matrix X(I, J), whose (i, j) entry X i,j , i < j, is given as follows: (3) If k + 1 ≤ i < j ≤ n, then We put (4.7) By pulling out the factor f (x −1 i ) from the ith row/column with i ∈ J and f (x i ) from the ith row/column with i ∈ [k] \ J, and then by multiplying the last (n − k) rows/columns by −1, we obtain where the entries X i,j of the skew-symmetric matrix X (I, J) are given by Let σ be the ring automorphism of the Laurent polynomial ring Then, by using we have Hence σ Pf X (I, J) = Pf σ(X i,j ) 1≤i,j≤n can be evaluated by the following lemma. , let Y (K) be the n × n skew-symmetric matrix with (i, j) entry given by if i ∈ K and j ∈ K.
Then we have Pf Y (K) = (−1) Σ(K) (i,j)∈D + n (K) where Σ(K) = i∈K i and Applying this lemma to K = C 1 ∪ C 4 ∪ C 5 , we have Then by using (4.9), we have , we see that Hence, combining these relations with (4.8) and (4.10), we obtain the following expression for the coefficient L(I, J) in the left hand side: Now we can finish the proof of Theorem 4.2.
Since n is even, we obtain (4.12). Therefore we have R(I, J) = L(I, J). This completes the proof of Theorem 4.2, and hence of Theorem 3.1.

Shifted plane partitions of double staircase shape
In this section we find generating functions of shifted plane partitions by specializing the variables in the character identities in Theorem 3.1. Also we derive Hopkins-Lai's formula for the number of lozenge tilings of flashlight regions.
Then, by specializing x i = q i or q i−1/2 in the character identities in Theorem 3.1, we obtain the generating functions for shifted plane partitions of shifted double staircase shape with respect to the weights v(σ) and w(σ) defined by (1.7) and (1.8) respectively.
Theorem 5.1. Suppose 0 ≤ k ≤ n, and let a and m be nonnegative integers.

Proof of Theorem 5.1
In this subsection we derive Theorem 5.1 from Theorem 3.1.
Since these procedures are invertible, the correspondence σ → T gives a bijection between A(S(δ n + δ k ); λ) and SpTab (k,n−k) (λ). And, since the multiplicity of l in π is equal to the difference of the traces t l−1 (σ) − t l (σ), where t l (σ) is defined by (1.9) and t n+k (σ) = 0, the multiplicity m T (γ) of γ in T is given by Then we can show which imply (5.9). It follows from the Jacobi-Trudi-type identity (2.11) that . , x 1 |x n , . . . , x k+1 ).
In order to derive Theorem 5.1, we need the following formulas for the specializations of classical group characters corresponding to rectangular partitions.
Now we are ready to prove Theorem 5.1.
we can arrive at the expression (5.5).

Lozenge tilings of flashlight regions
In this subsection, we apply the character identity (Theorem 3.1 (1)) to enumerate the lozenge tilings of flashlight regions in the triangular lattice. A lozenge is the union of two unit equilateral triangles joined along an edge. A lozenge tiling of a region R in the regular triangular lattice is a covering of R by lozenges with neither gap nor overlap. For nonnegative integers x, y, z and t, let F x,y,z,t be the "flashlight region" x + y + z y t 2z x + t y + 2z  Figure 2, where the dashed line indicates a free boundary, i.e., lozenges are allowed to protrude across it. We consider lozenge tilings of F x,y,z,t by three types of lozenges given in Figure 3. Then Hopkins and Lai [13] obtain the following product formula for the number of lozenge tilings of F x,y,z,t , and use the case t = 0 to derive the formula for the number of shifted plane partitions of shifted double staircase shape (Theorem 1.1).
Theorem 5.4. ([13, Theorem 1.2] for y > 0) For nonnegative integers x, y, z and t, the number M (F x,y,z,t ) of lozenge tilings of the region F x,y,z,t is given by The proof by Hopkins-Lai [13] is based on an extension of Kuo condensation to regions with a free boundary due to Ciucu [7]. Here we use the character identity in Theorem 3.1 (1) to give an alternate proof.
In order to apply the character identity, we give an interpretation of intermediate symplectic characters in terms of lozenge tilings. Let 0 ≤ k ≤ n. Given a partition λ such that l(λ) ≤ n and λ 1 ≤ m, we denote by R (k,n−k) m (λ) the region obtained from F m,n−k,k,0 by adjoining n left-pointing triangles at the positions λ 1 + n − 1, . . . , λ n−1 + 1, λ n to the free boundary of F m,n−k,k,0 , where the vertical edges on the free boundary are labeled with 0, 1, . . . , m + n − 1 from bottom to top. For example, the region R (2,2) 4 (4, 3, 1, 1) is depicted in Figure 4. We denote by T (R of R (k,n−k) m (λ), or F m,n−k,k,a , we define its weight wt(T ) as follows. We assign to each SW-NE lozenge L in the ith column from the right a weight and put wt(L) = 1 for the other two types of lozenges L. Then the weight wt(T ) of a tiling T is defined as the product of all the weights of the lozenges used in the tiling T . For example, if T is the tiling given in the left picture of Figure 5, then its weight is computed as The following lemma provides an interpretatioin of sp  Proof. By Lemma 5.2, we have a bijection between SpTab (k,n−k) (λ) and A m (S(δ n + δ k ); λ). And there is a natural bijection between T (R (k,n−k) m (λ)) and A m (S(δ n + δ k ); λ), which is obtained by reading the "height" of the horizontal lozenges along the paths consisting of horizontal and SW-NE lozenges. The desired bijection is obtained by composing these two bijections. See Figure 5. Now Theorem 5.4 follows from Theorem 3.1 (1) by using Lemma 5.5.
Proof of Theorem 5.4. Let M m,n−k,k,a = M m,n−k,k,a (x 1 , . . . , x k |x k+1 , . . . , x n ) be the generating function of tilings of F m,n−k,k,a . We prove M m,n−k,k,a = o B (m/2) n (x 1 , . . . , x n ) · sp (m/2+a) k (x 1 , . . . , x k ) · (x k+1 · · · x n ) m/2 . By specializing x 1 = · · · = x n = 1 in (5.19) and using (5.12), (5.13) with q = 1, we obtain the desired identity (5.18). Let F m,n−k,k,a be the region obtained from F m+a,n−k,k,0 by changing a edges from the bottom on the free boundary into a non-free boundary (see Figure 6), and M m,n−k,k,a the m + n a n − k 2k m + a n + k Figure 6: The region F m,n−k,k,a for m = 4, n − k = 4, k = 3, a = 2 tiling generating function of F m,n−k,k,a . Because of the presence of the non-free boundary, the bottom-left corner (the shaded region in Figure 6) is tiled by forced SW-NE lozenges, and we have M m,n−k,k,a = (x k+1 · · · x n ) a · M m,n−k,k,a .
Hence Equation ( (1 r ) for r ≥ 0 and put e To state dual Jacobi-Trudi formulas, we introduce some notations. We define e • r (x ±1 1 , . . . , x ±1 k ) by the generating function With these notations we have the following dual Jacobi-Trudi formulas.
Proposition A.3. Let λ be a partition of length ≤ n and λ the conjugate partition of λ. In the proof of Proposition A.3, we use the following relations: