Xrfgtjtk

Suppose $X$ and $Y$ are independent sub-Gaussian random variables with 0 mean and $sigma^2$ sub-Gaussian parameter. More specifically, $mathbb E[exp(a^T X)]leq exp{|a|_2^2sigma^2/2}$ for all $a$, and the same holds for $Y$ as well.

I wish to upper bound the tail probability
$$
Prleft[|X^T Y|>tright]
$$
using $sigma^2$ and dimension $n$ (that is, both $X$ and $Y$ are $d$-dimensional random variables). How can I achieve this? $X^T Y$ does not seem to be either sub-Gaussian or sub-exponential.

Though you are right that the product is in general not sub-Gaussian, but it is still subexponential.

Note that for a sub-Gaussian vector $X$ in $mathbb{R}^n$ with parameter $sigma^2$ and for $r<sigma^2$,
$$
mathrm{E}bigg[expBig{frac{r||X||^2}2Big}bigg] = frac{1}{(2pi r)^{n/2}}int_{mathrm{R}^n} expBig{-frac{||a||^2}{2r}Big}, mathbb{E}[e^{a^T X}]da \
le frac{1}{(2pi r)^{n/2}}int_{mathrm{R}^n} expBig{-frac{||a||^2}{2r} + frac{||a^2||sigma^2}{2}Big}da = frac{1}{big(2pi (sigma^2-r)big)^{n/2}}.
$$
Therefore, thanks to independence, for any $lambda<1$,
$$
mathrm{E}big[exp{lambda X^T Y}big] le mathrm{E}bigg[expBig{frac{lambda^2sigma^2||X||^2}2Big}bigg]lefrac{1}{big(2pi sigma^2(1-lambda^2)big)^{n/2}}.
$$
Hence you can easily derive an exponential bound for $mathrm{P}(X^TY>t)$.

By Cauchy-Schwarz, the event $[ |X^TY|>t]$ is a subset of $[| X|>sqrt t] cup[|Y|>sqrt t]$, so its probability is bounded above by $P(| X|>sqrt t)+P(| Y|>sqrt t)$, for which $O(exp{-At})$ bounds can probably be derived, for some $sigma$-dependent $A$.

Moment generating functions of subgaussian vector can often be bounded from above by the same moment generating function, with the subgaussian vector replaced by a standard normal. This is equivalent to zhoraster's answer.

Take $sigma=1$ without loss of generality (otherwise, consider $X/sigma$ and $Y/sigma$ and scale back afterwards).
Let $g,h$ be standard normal random vectors, independent of $X,Y$. Then by independence,
[
E[e^{uX^TY} | X ] = E[e^{u^2|X|^2/2} ] = E[e^{u X^top g}].
]
Now by the law of total expectation, and a similar argument for $Y$,
[
E[e^{uX^TY}] le E[ E[e^{u X^T g}| g] ] = E[E[ e^{u^2|g|^2/2} | g]] = E[e^{u g^Th}].
]
So the problem is reduced to bounding the moment generating function where both random vectors are standard normal.