Xrfgtjtk

What are the coordinates and the radius of the incircle defined by three lines expressed in normal form?

The lines are $a_i x + b_i y + c_i = 0$, where $a_i^2 + b_i^2 = 1$ and $i = 1, 2, 3$.

The coordinates of the incircle are $(x_c, y_c)$ and the radius is $r_c$.

I found a solution based on finding the vertices of the triangle formed by the three lines expressed in standard form, but I suspect that it should be a simpler solution using the normal form of the lines.

One approach is to work a bit backwards.

Consider the three lines with normal form
$$begin{align}
x cos 2alpha + y sin 2alpha &= r \
x cos 2beta + y sin 2beta &= r \
x cos 2gamma + y sin 2gamma &= r
end{align} tag{1}$$
The incircle of the corresponding triangle has radius $r$ and center $(0,0)$.

We can get three lines whose incircle has radius $r$ and whose center is $(h,k)$ by translating the lines in $(1)$ via the substitution: $xto x-h$, $yto y-k$. We can write the results as

$$begin{align}
x cos 2alpha + y sin 2alpha &= r + h cos 2alpha + k sin 2alpha \
x cos 2beta, + y sin 2beta, &= r + h cos 2beta + k sin 2beta\
x cos 2gamma, + y sin 2gamma, &= r + h cos 2gamma + k sin 2gamma
end{align} tag{2}$$

Conveniently, these equations are already in normal form. If we assign $a$, $b$, $c$ to the "constant terms" (that is, the right-hand sides) of the equations in $(2)$, we have a linear system in $r$, $h$, $k$ that we can readily solve (via linear combinations, Cramer's Rule, or what-have-you). The result reduces with a bit of trig, and we find

The incircle of the triangle with side-lines
$$begin{align}
x cos 2alpha + y sin 2alpha &= a \
x cos 2beta, + y sin 2beta, &= b\
x cos 2gamma, + y sin 2gamma, &= c
end{align} tag{3}$$
has center $(h,k)$ and radius $r$, where
$$begin{align}
h &= phantom{-}frac{1}{2sigma}left(;
a cos(beta + gamma) sin(beta - gamma)
+ b cos(gamma + alpha) sin(gamma - alpha)
+ c cos(alpha + beta) sin(alpha - beta);right) \[4pt]
k &= phantom{-}frac{1}{2sigma}left(; a sin(beta + gamma) sin(beta - gamma) +
b sin(gamma + alpha) sin(gamma-alpha) + c sin(alpha+beta) sin(alpha-beta);right) \[4pt]
r &= -frac{1}{4sigma}left(;a sin 2 (beta - gamma) + b sin 2 (gamma - alpha) + c sin 2 (alpha - beta);right) \[4pt]
sigma &= phantom{-}sin(beta-gamma)sin(gamma - alpha) sin(alpha - beta) end{align}$$

Assuming none of the lines are parallel, the center of a circle which is tangent to all three lines at once will necessarily lie on the intersection of the angle bisectors of where the lines intersect. So let's find those bisectors.

For this to work nicely, we first need to normalize the three lines (i.e. multiply or divide each equation by separate constants so that $a_1^2 + b_1^2 = a_2^2 + b_2^2 = a_3^2 + b_3^2$). You have already done this, but it's important not to forget, so I mention it here.

Take the intersection of lines 1 and 2. The two angle bisectors of that intersection are
$$
a_1x + b_1y + c_1 = a_2x + b_2y + c_2\ tag{1}
a_1x + b_1y + c_1 = -a_2x - b_2y - c_2
$$
Similarily, we get that the two bisectors at the intersection of lines 2 and 3 are given by
$$
a_2x + b_2y + c_2 = a_3x + b_3y + c_3\tag2
a_2x + b_2y + c_2 = -a_3x - b_3y - c3
$$
(We don't need the bisectors of the intersection between 1 and 3, as that doesn't give us any new information).

Now, pick one of the two lines from $(1)$ and one of the lines from $(2)$, and find their intersection. Then do the same for the three other combinations. What you now have are the centers of the incircle and the three excircles. The one with the smallest radius is the incircle.

The incircle center coordinates $q$ can be obtained solving a minimization problem.

Given the lines

$$
l_ito p = p_i + lambda_ivec v_i, i = 1,2,3
$$

minimizing

$$
D(q,lambda) = sum_{i}^3||p_i+lambda_ivec v_i-q||^2
$$

Now solving

$$
D_q = sum_i^3 p_i+lambda_i vec v_i - q=vec 0\
D_{lambda_j} = sum_i^3 p_icdot vec v_j + lambda_i vec v_icdot vec v_j -qcdot vec v_j = 0
$$

which is a linear system in $q,lambda$ we obtain the incenter $q^*$ and the tangency points $p_i+lambda_i^*vec v_i$

or after substitutions

$$
left(begin{array}{ccc}
vec v_1cdotvec v_1 & vec v_1cdotvec v_2 & vec v_1cdotvec v_3\
vec v_2cdotvec v_1 & vec v_2cdotvec v_2 & vec v_2cdotvec v_3\
vec v_3cdotvec v_1 & vec v_3cdotvec v_2 & vec v_3cdotvec v_3
end{array}right)left(begin{array}{c}lambda_1\ lambda_2\ lambda_3end{array}right) = left(begin{array}{c}(p_1+p_2+p_3)cdot vec v_1\ (p_1+p_2+p_3)cdot vec v_2\ (p_1+p_2+p_3)cdot vec v_3end{array}right)
$$

Now considering that $vec v_1cdotvec v_j = a_i a_j+b_i b_j$ and imposing $a_i^2+b_i^2 = 1$ we have

$$
M = left(begin{array}{ccc}
vec v_1cdotvec v_1 & vec v_1cdotvec v_2 & vec v_1cdotvec v_3\
vec v_2cdotvec v_1 & vec v_2cdotvec v_2 & vec v_2cdotvec v_3\
vec v_3cdotvec v_1 & vec v_3cdotvec v_2 & vec v_3cdotvec v_3
end{array}right) = left(begin{array}{ccc}
1 & cos u & cos v\
cos u & 1 & cos w\
cos v & cos w & 1
end{array}right)
$$

and then

$$
M^{-1} = frac{1}{Delta}left(
begin{array}{ccc}
sin ^2(w) & cos (v) cos (w)-cos (u) & cos (u) cos (w)-cos (v) \
cos (v) cos (w)-cos (u) & sin ^2(v) & cos (u) cos (v)-cos (w) \
cos (u) cos (w)-cos (v) & cos (u) cos (v)-cos (w) & sin ^2(u) \
end{array}
right)
$$

with $Delta = det M = 2 cos (u) cos (v) cos (w)-cos ^2(u)-cos ^2(v)+sin ^2(w)$

NOTE

$$
a_i x + b_i y + c_i = 0equiv ((x,y)-(0,-frac{c_i}{b_i}))cdot(a_i,b_i)=0equiv p = p_i+lambda_i vec v_i
$$

with

$$
p = (x,y)\
p_i = (0,-frac{c_i}{b_i})\
vec v_i = (a_i, b_i)
$$