Back propagation equation proof












1












$begingroup$


I am trying to prove this equation (from the backpropagation equations in AI).



$$frac{partial C}{partial b_j^l} = delta_j^l$$



C is the cost function: $C = frac{1}{2}||y - a^L||^2$



Where the output of layer l and neuron j is express like so $a^l_j=σ(∑_kw^l_{jk}a^{l−1}_k+b^l_j)$



I am suppose to use this assertion to do the demonstration: $delta_j^L = frac{partial C}{partial z_j^L}$



So far, here is what I have tried:



$$frac{partial C}{partial z_j^L} = sum_k frac{partial C}{partial z_j^L} frac{partial b_j^l}{partial b_j^L} $$ (I am using the chain rule to have a sum)



<=> $$frac{partial C}{partial z_j^L} = sum_k frac{partial C}{partial b_j^l} frac{partial b_j^l}{partial z_j^L} $$



So I guess I have to prove, that $frac{partial b_j^l}{partial z_j^L}$ equals 1.



But I don't have any ideas how to prove it.



Thanks for your help



N.B
I am following this course => http://neuralnetworksanddeeplearning.com/chap2.html where the first two equations of the BackPropagation equations are already proved, and the 2 others should be proved the same way (using the chain rule)






artificial-intelligence







share|cite|improve this question









$endgroup$

















    1












    $begingroup$


    I am trying to prove this equation (from the backpropagation equations in AI).



    $$frac{partial C}{partial b_j^l} = delta_j^l$$



    C is the cost function: $C = frac{1}{2}||y - a^L||^2$



    Where the output of layer l and neuron j is express like so $a^l_j=σ(∑_kw^l_{jk}a^{l−1}_k+b^l_j)$



    I am suppose to use this assertion to do the demonstration: $delta_j^L = frac{partial C}{partial z_j^L}$



    So far, here is what I have tried:



    $$frac{partial C}{partial z_j^L} = sum_k frac{partial C}{partial z_j^L} frac{partial b_j^l}{partial b_j^L} $$ (I am using the chain rule to have a sum)



    <=> $$frac{partial C}{partial z_j^L} = sum_k frac{partial C}{partial b_j^l} frac{partial b_j^l}{partial z_j^L} $$



    So I guess I have to prove, that $frac{partial b_j^l}{partial z_j^L}$ equals 1.



    But I don't have any ideas how to prove it.



    Thanks for your help



    N.B
    I am following this course => http://neuralnetworksanddeeplearning.com/chap2.html where the first two equations of the BackPropagation equations are already proved, and the 2 others should be proved the same way (using the chain rule)






    artificial-intelligence







    share|cite|improve this question









    $endgroup$















      1












      1








      1


      0



      $begingroup$


      I am trying to prove this equation (from the backpropagation equations in AI).



      $$frac{partial C}{partial b_j^l} = delta_j^l$$



      C is the cost function: $C = frac{1}{2}||y - a^L||^2$



      Where the output of layer l and neuron j is express like so $a^l_j=σ(∑_kw^l_{jk}a^{l−1}_k+b^l_j)$



      I am suppose to use this assertion to do the demonstration: $delta_j^L = frac{partial C}{partial z_j^L}$



      So far, here is what I have tried:



      $$frac{partial C}{partial z_j^L} = sum_k frac{partial C}{partial z_j^L} frac{partial b_j^l}{partial b_j^L} $$ (I am using the chain rule to have a sum)



      <=> $$frac{partial C}{partial z_j^L} = sum_k frac{partial C}{partial b_j^l} frac{partial b_j^l}{partial z_j^L} $$



      So I guess I have to prove, that $frac{partial b_j^l}{partial z_j^L}$ equals 1.



      But I don't have any ideas how to prove it.



      Thanks for your help



      N.B
      I am following this course => http://neuralnetworksanddeeplearning.com/chap2.html where the first two equations of the BackPropagation equations are already proved, and the 2 others should be proved the same way (using the chain rule)






      artificial-intelligence







      share|cite|improve this question









      $endgroup$




      I am trying to prove this equation (from the backpropagation equations in AI).



      $$frac{partial C}{partial b_j^l} = delta_j^l$$



      C is the cost function: $C = frac{1}{2}||y - a^L||^2$



      Where the output of layer l and neuron j is express like so $a^l_j=σ(∑_kw^l_{jk}a^{l−1}_k+b^l_j)$



      I am suppose to use this assertion to do the demonstration: $delta_j^L = frac{partial C}{partial z_j^L}$



      So far, here is what I have tried:



      $$frac{partial C}{partial z_j^L} = sum_k frac{partial C}{partial z_j^L} frac{partial b_j^l}{partial b_j^L} $$ (I am using the chain rule to have a sum)



      <=> $$frac{partial C}{partial z_j^L} = sum_k frac{partial C}{partial b_j^l} frac{partial b_j^l}{partial z_j^L} $$



      So I guess I have to prove, that $frac{partial b_j^l}{partial z_j^L}$ equals 1.



      But I don't have any ideas how to prove it.



      Thanks for your help



      N.B
      I am following this course => http://neuralnetworksanddeeplearning.com/chap2.html where the first two equations of the BackPropagation equations are already proved, and the 2 others should be proved the same way (using the chain rule)






      artificial-intelligence




      artificial-intelligence






      share|cite|improve this question













      share|cite|improve this question











      share|cite|improve this question




      share|cite|improve this question










      asked Nov 18 '18 at 13:41









      UnepierreUnepierre

      62




      62






















          1 Answer
          1






          active

          oldest

          votes


















          0












          $begingroup$

          By definition, $$ frac{partial C}{partial z_j^ell} = delta_j^ell $$ $$ z^ell_k = sum_i w_{ki}^ell a^{ell -1}_j + b_k^{ell} $$
          the latter of which means that
          $$ frac{partial z^ell_k}{partial b_alpha^ell} = Delta^ell_{kalpha}=begin{cases}
          1 & text{if } k=alpha \ 0 & text{otherwise} end{cases} $$
          We want to show that
          $$ frac{partial C}{partial b_j^ell} = delta_j^ell $$
          Since $C$ depends on $b_beta$ through $z^beta$, we can use the chain rule:
          begin{align}
          frac{partial C}{partial b_j^ell}
          &= sum_zeta frac{partial C}{partial z_zeta^ell} frac{partial z_zeta^ell}{partial b_j^ell} \
          &= sum_zeta delta^ell_zeta Delta^ell_{zeta j} \
          &= delta^ell_j
          end{align}
          where the Kronecker delta $Delta_{zeta j}^ell$ collapses out the sum except for the $j$th term.






          share|cite|improve this answer









          $endgroup$














            Your Answer








            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "69"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            noCode: true, onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3003546%2fback-propagation-equation-proof%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            0












            $begingroup$

            By definition, $$ frac{partial C}{partial z_j^ell} = delta_j^ell $$ $$ z^ell_k = sum_i w_{ki}^ell a^{ell -1}_j + b_k^{ell} $$
            the latter of which means that
            $$ frac{partial z^ell_k}{partial b_alpha^ell} = Delta^ell_{kalpha}=begin{cases}
            1 & text{if } k=alpha \ 0 & text{otherwise} end{cases} $$
            We want to show that
            $$ frac{partial C}{partial b_j^ell} = delta_j^ell $$
            Since $C$ depends on $b_beta$ through $z^beta$, we can use the chain rule:
            begin{align}
            frac{partial C}{partial b_j^ell}
            &= sum_zeta frac{partial C}{partial z_zeta^ell} frac{partial z_zeta^ell}{partial b_j^ell} \
            &= sum_zeta delta^ell_zeta Delta^ell_{zeta j} \
            &= delta^ell_j
            end{align}
            where the Kronecker delta $Delta_{zeta j}^ell$ collapses out the sum except for the $j$th term.






            share|cite|improve this answer









            $endgroup$


















              0












              $begingroup$

              By definition, $$ frac{partial C}{partial z_j^ell} = delta_j^ell $$ $$ z^ell_k = sum_i w_{ki}^ell a^{ell -1}_j + b_k^{ell} $$
              the latter of which means that
              $$ frac{partial z^ell_k}{partial b_alpha^ell} = Delta^ell_{kalpha}=begin{cases}
              1 & text{if } k=alpha \ 0 & text{otherwise} end{cases} $$
              We want to show that
              $$ frac{partial C}{partial b_j^ell} = delta_j^ell $$
              Since $C$ depends on $b_beta$ through $z^beta$, we can use the chain rule:
              begin{align}
              frac{partial C}{partial b_j^ell}
              &= sum_zeta frac{partial C}{partial z_zeta^ell} frac{partial z_zeta^ell}{partial b_j^ell} \
              &= sum_zeta delta^ell_zeta Delta^ell_{zeta j} \
              &= delta^ell_j
              end{align}
              where the Kronecker delta $Delta_{zeta j}^ell$ collapses out the sum except for the $j$th term.






              share|cite|improve this answer









              $endgroup$
















                0












                0








                0





                $begingroup$

                By definition, $$ frac{partial C}{partial z_j^ell} = delta_j^ell $$ $$ z^ell_k = sum_i w_{ki}^ell a^{ell -1}_j + b_k^{ell} $$
                the latter of which means that
                $$ frac{partial z^ell_k}{partial b_alpha^ell} = Delta^ell_{kalpha}=begin{cases}
                1 & text{if } k=alpha \ 0 & text{otherwise} end{cases} $$
                We want to show that
                $$ frac{partial C}{partial b_j^ell} = delta_j^ell $$
                Since $C$ depends on $b_beta$ through $z^beta$, we can use the chain rule:
                begin{align}
                frac{partial C}{partial b_j^ell}
                &= sum_zeta frac{partial C}{partial z_zeta^ell} frac{partial z_zeta^ell}{partial b_j^ell} \
                &= sum_zeta delta^ell_zeta Delta^ell_{zeta j} \
                &= delta^ell_j
                end{align}
                where the Kronecker delta $Delta_{zeta j}^ell$ collapses out the sum except for the $j$th term.






                share|cite|improve this answer









                $endgroup$



                By definition, $$ frac{partial C}{partial z_j^ell} = delta_j^ell $$ $$ z^ell_k = sum_i w_{ki}^ell a^{ell -1}_j + b_k^{ell} $$
                the latter of which means that
                $$ frac{partial z^ell_k}{partial b_alpha^ell} = Delta^ell_{kalpha}=begin{cases}
                1 & text{if } k=alpha \ 0 & text{otherwise} end{cases} $$
                We want to show that
                $$ frac{partial C}{partial b_j^ell} = delta_j^ell $$
                Since $C$ depends on $b_beta$ through $z^beta$, we can use the chain rule:
                begin{align}
                frac{partial C}{partial b_j^ell}
                &= sum_zeta frac{partial C}{partial z_zeta^ell} frac{partial z_zeta^ell}{partial b_j^ell} \
                &= sum_zeta delta^ell_zeta Delta^ell_{zeta j} \
                &= delta^ell_j
                end{align}
                where the Kronecker delta $Delta_{zeta j}^ell$ collapses out the sum except for the $j$th term.







                share|cite|improve this answer












                share|cite|improve this answer



                share|cite|improve this answer










                answered Jan 13 at 22:52









                user3658307user3658307

                5,0833949




                5,0833949






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Mathematics Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    Use MathJax to format equations. MathJax reference.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3003546%2fback-propagation-equation-proof%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    -

                    Popular posts from this blog

                    Måne

                    Image
                    För jordens måne, se Månen. Några månar i solsystemet i skala till vår måne En måne , naturlig satellit eller drabant är en himlakropp som kretsar kring en planet eller en asteroid i ett solsystem. Ordet i bestämd form – månen – avser planeten jordens måne. Det finns inte i något fall belagt att någon måne har egna månar. De flesta stora månar i solsystemet är bundna i synkron rotation med en och samma sida vänd mot planeten. Ett exempel på en måne som inte beter sig på detta vis, är Hyperion som kaotiskt roterar kring Saturnus. En asteroidmåne är en måne som kretsar kring en asteroid. En måne kan ha bildats samtidigt som moderplaneten eller senare infångats av planetens gravitationskraft. I vårt eget solsystem känner vi i nuläget till 173 månar runt sex av de åtta planeterna [ 1 ] : Jorden – månen Mars – Phobos och Deimos Jupiter – Ganymedes (som är solsystemets största måne), Io, Europa, Callisto och 59 andra kända månar Saturnus – Titan (den enda måne...
                    Read more

                    Storängen

                    Image
                    För stadsdelen i Mariehamn, se Storängen, Åland. För ett naturreservat i Norrtälje kommun, se Storängens naturreservat. Storängen från luften i september 1967, vy mot väst. Till vänster syns Järlasjön och Saltsjöbanans spår. Till höger löper Värmdövägen och längst ut till höger syns anläggningsarbetena för Värmdöleden. Storängen är ett villaområde i centrala Nacka i Stockholms län. Området är beläget strax sydost om Nacka Forum vid Saltsjöbanan mellan Lillängen och Saltsjö-Duvnäs. Storängens villasamhälle grundades 1904 av "Tjänstemännens Egnahemsförening vid Storängen" och de första villorna började byggas samma år. Redan 1909 var 107 tomter bebyggda av totalt runt 160. [ 1 ] År 2013 fanns drygt 150 villor med lika många familjer i området. Egnahemsföreningen existerar fortfarande och fungerar idag som intresseförening för de boende i Storängen. Storängen anses vara ett av Sveriges bäst bevarade villasamhällen från 1900-talets början, tack vare sina tidstypis...
                    Read more

                    VLT Carioca

                    Image
                    VLT Carioca Informações Proprietário Prefeitura do Rio de Janeiro Local [[Rio de Janeiro, RJ]] País {{country data Brasil| country flag2| name = Brasil| variant = | size = }} Tipo de transporte Bonde (Veículo leve sobre trilhos) Número de linhas 2 (+1 em construção) Número de estações 26 (+3 em construção) Tráfego 80 mil passageiros/dia [ 1 ] (dezembro/2018) Website www.vltrio.com.br Funcionamento Início de funcionamento 5 de junho de 2016 (2 anos) Operadora(s) Concessionária do VLT Carioca S.A. Número de veículos 32 Alstom Citadis Comprimento dos veículos 44 m (144 ft) Dados técnicos Extensão do sistema 28 km (17,4 mi) [ 2 ] Frequência 5–10 minutos Bitola Bitola padrão/internacional 1 435 mm (4,71 ft) Eletrificação APS, 750V DC Velocidade média 15 km/h (9,32 mph) [ 2 ] Velocidade máxima 50 km/h (31,1 mph) ...
                    Read more