So, I studied linear algebra back in college and those equations really caught my attention. Here's what I got from it:

For every tape m2 and tape m2' so that (s.t.) the function of both tapes is equal, we have that the function of both subtracted is null. This means the function is an homomorphism.

I got three things from this:
1. the correction the student made at the beginning was correct, and, thus, the professor didn't realize he had made a mistake (might be symbolic of the plot itself);
2. The function itself is symbolic of the plot, since "homomorphism" means the properties of the ring*, or in this case curse, are always preserved after an operation with the tapes (such as copying) and
3. m2-m2' = 0, so m2=m2', so both tapes are identical and by subtracting one from the other we get zero, i.e. no curse.

The next line is confusing, but I believe it means it's possible to get to yet another homomorphic function from the previous one, which would mean another cycle of the curse from copying the previous tapes.

Next we get that this new new function is injective, which in this ring*, means the previous ones were like that as well. Injective means applying the function for two different values will never yield the same result.

Here's why this is interesting: while this means you'll could never copy any other film in the world, such as Titanic, for example, and put the curse on it; there's also another, more interesting implication, that all the tapes came from a previous source, INCLUDING the original one we see.
Now, this could be the kid's TV, or sadako's hate itself, both mentioned in the movie.

And that's exactly the conclusion he arrived at the last line of the chalkboard.
What the hell, I might just be right about this one.

*by the way, in math a "ring" is an algebraic structure consisting of a set together with two binary operations usually called addition and multiplication, which are the only two we see written on the board. So he was working on a ringu the whole time.